41. tcpdump and BPF filters

41.1 Anatomy of a capture command

sudo tcpdump -i eth0 -nn -vv -e -s0 -c 200 -w /tmp/cap.pcap 'tcp port 443 and host 93.184.216.34'
            +iface+  |   |   |  |   |   |   +-- output file       +------ BPF filter ------+
                     |   |   |  |   |   +-- stop after N packets
                     |   |   |  |   +-- full packet (default is full on modern tcpdump)
                     |   |   |  +-- print link-layer (MAC) header
                     |   |   +-- extra verbose (TTL, IP id, options)
                     |   +-- don't resolve hostnames (-n) or ports (-nn); avoids DNS stalls

Useful flags: -A (payload as ASCII), -X (hex+ASCII), -tttt (readable timestamps), -Q in|out (direction), --time-stamp-precision=nano, -G 60 -w cap-%H%M%S.pcap (rotate hourly), -C 100 (rotate at 100 MB), -Z user (drop privileges).

Golden rules

• Always -w to a file for anything you'll analyze; read later with -r or in Wireshark. Printing to the terminal drops packets under load.
• -s0 + high rate can overflow the kernel capture buffer → "N packets dropped by kernel". Raise -B 4096 (KiB) or snap shorter (-s 128 for headers only).
• Filter as tightly as possible at capture time; CPU and disk are finite.
• Capture on both ends for anything involving loss/latency — you can only prove "sent here, not received there" with two captures.

41.2 BPF filter cookbook

host 10.0.0.5                       # to or from
src host / dst host 10.0.0.5
net 10.0.0.0/24
port 443            portrange 8000-8100
tcp port 443 and not host 10.1.1.1

# TCP flags (tcp[tcpflags] is a bitmask):
'tcp[tcpflags] & (tcp-syn) != 0'                 # any SYN (incl SYN-ACK)
'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'   # SYN only (connection attempts)
'tcp[tcpflags] & (tcp-rst) != 0'                 # resets
'tcp[tcpflags] & (tcp-fin) != 0'                 # FINs
'(tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst)) != 0'   # setup/teardown only

# Payload / size:
'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'  # TCP payload starts with "GET "
'ip[2:2] > 1400'                                  # IP total length > 1400
'greater 1400'
'ip[6:2] & 0x3fff != 0'                           # IPv4 fragments (MF set or offset != 0)
'tcp[13] & 32 != 0'                               # URG flag set (weird; middlebox check)

# Other:
'icmp or icmp6'
'arp'
'vlan 100'
'udp port 53'
'ip proto 47'                                     # GRE
'ether host aa:bb:cc:dd:ee:ff'

tcpdump filters are compiled BPF and run in the kernel — very cheap. Display filters (Wireshark's tcp.analysis.retransmission etc.) run in userspace on the already-captured data and are far more expressive; use capture filters to bound volume, display filters to investigate.

41.3 Reading tcpdump TCP output

12:00:00.100000 IP 10.0.0.5.51000 > 93.184.216.34.443: Flags [S], seq 1000,
                win 64240, options [mss 1460,sackOK,TS val 1 ecr 0,nop,wscale 7], length 0
12:00:00.190000 IP 93.184.216.34.443 > 10.0.0.5.51000: Flags [S.], seq 500, ack 1001,
                win 65535, options [mss 1460,sackOK,TS val 9 ecr 1,nop,wscale 9], length 0
12:00:00.190100 IP 10.0.0.5.51000 > 93.184.216.34.443: Flags [.], ack 501, win 502, length 0
12:00:00.190200 IP 10.0.0.5.51000 > 93.184.216.34.443: Flags [P.], seq 1001:1518, ack 501,
                win 502, length 517           # TLS ClientHello (517 bytes)

• Flags [S] SYN, [S.] SYN-ACK, [.] bare ACK, [P.] PSH+ACK (data), [F.] FIN, [R] RST.
• seq 1001:1518 = bytes 1001..1517 (length 517). Relative numbers by default.
• win 502 after the handshake = scaled (502 × 2^7 ≈ 64 KB).
• ack 501 = "I have everything through 500."
• options [nop,nop,sack 1 {3001:4001}] on a later ACK = a SACK block.
• ecr echoes the peer's val — that pair is your RTT sample source.

41.4 War story

"Packet loss between app and DB" — a single capture on the app server showed retransmissions. Capturing simultaneously on the DB showed the "lost" segments did arrive, were ACKed, but the ACKs never reached the app. The loss was one-directional on the return path through a flaky LACP member. A single capture would have blamed the wrong direction. Lesson: for loss/latency, capture both ends with synchronized clocks (-tttt, or NTP/PTP), then compare by sequence number.

Try it yourself

sudo tcpdump -i any -nn -c20 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'
sudo tcpdump -i eth0 -nn -s128 -w /tmp/db.pcap -B 8192 'host 10.2.0.9 and tcp port 5432'
sudo tcpdump -r /tmp/db.pcap -nn -ttt 'tcp[tcpflags] & tcp-rst != 0'
# ring buffer for an intermittent issue (24 files x 100MB):
sudo tcpdump -i eth0 -nn -s0 -C 100 -W 24 -w /var/tmp/ring.pcap 'port 443'