Part 8 — Seeing and fixing the network

Networking is the most observable subject in computing: you can watch every byte. This part turns you from someone who guesses into someone who looks.

Chapter 31 — `tcpdump`: watching packets go by

In one sentence

tcpdump prints every packet matching a filter, and once you can read its output you can answer questions no log file can.

The anatomy of a command

   sudo tcpdump -i any -n -c 20 'tcp port 443 and host example.com'
        \______/ \____/ \/ \___/ \_______________________________/
           |      which  no  stop            the FILTER
        needs   interface DNS  after         (in quotes)
        root              lookups 20

Flags you'll use constantly:

Flag	Does
`-i any`	capture on all interfaces (or `-i eth0` for one)
`-n`	don't resolve hostnames — always use this; DNS lookups slow you down and pollute the capture
`-nn`	also don't resolve port names (shows `443` not `https`)
`-c 20`	stop after 20 packets
`-e`	show the Ethernet (MAC) header
`-A`	show the payload as text (great for HTTP)
`-X`	show payload as hex + text
`-w file.pcap`	write to a file for Wireshark
`-r file.pcap`	read a saved file
`-t` / `-tttt`	timestamp formats (`-tttt` is human-readable)
`-s 0`	capture the whole packet (default on modern versions)

The filters you actually need

host 192.168.1.50              # to or from this address
src host 192.168.1.50          # only from
dst host 192.168.1.50          # only to
net 192.168.1.0/24             # a whole network
port 443                       # to or from this port
portrange 8000-8100
tcp / udp / icmp / arp         # by protocol
tcp port 443 and host example.com          # combine with and / or / not
not port 22                    # exclude your own SSH session (do this on servers!)

# TCP flags -- extremely useful:
'tcp[tcpflags] & (tcp-syn) != 0'                # any SYN (connection attempts)
'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'  # SYNs only, not SYN-ACKs
'tcp[tcpflags] & (tcp-rst) != 0'                # resets (refused/aborted)
'tcp[tcpflags] & (tcp-fin) != 0'                # connection closes

Reading the output

14:32:07.891234 IP 192.168.1.50.51000 > 93.184.216.34.443: Flags [S],
      seq 1234567, win 64240, options [mss 1460,sackOK,wscale 7], length 0
\____________/    \_____________/   \______________/  \___/  \_______/
   timestamp        source            destination      flags   details
                  IP.port             IP.port

The flags shorthand (memorise this — it's most of reading tcpdump):

Symbol	Means
`[S]`	SYN — opening a connection
`[S.]`	SYN+ACK — accepting (the `.` means ACK)
`[.]`	ACK only — no data, just acknowledging
`[P.]`	PUSH+ACK — actual data
`[F.]`	FIN+ACK — closing politely
`[R]` / `[R.]`	RST — aborted / refused

So a healthy connection reads: [S] [S.] [.] [P.] [P.] ... [F.] [.]

The golden rules

Always use -n. Otherwise tcpdump does DNS lookups which generate more traffic which appears in your capture. Confusing and slow.
Filter tightly. On a busy machine, an unfiltered capture is unreadable and can drop packets.
On a remote server, exclude your own SSH session: and not port 22. Otherwise you capture yourself capturing, forever.
For anything you'll analyse, write to a file (-w) and open it in Wireshark. Terminal output is for quick looks.
To prove where packets are lost, capture on both ends simultaneously. A single capture can only tell you what arrived there.

Practice (35 min) — the essential recipes

A. A complete connection, start to finish:

sudo tcpdump -i any -n -tttt 'host example.com' -c 30
# in another terminal:
curl -s https://example.com > /dev/null

Identify: the DNS query, the handshake, the TLS negotiation, data, and the close.

B. All connection attempts leaving your machine:

sudo tcpdump -i any -n 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'

Leave it running. You will be surprised how much your machine talks to.

C. Find who's being refused:

sudo tcpdump -i any -n 'tcp[tcpflags] & tcp-rst != 0'

D. Read HTTP in plain text:

sudo tcpdump -i any -n -A 'tcp port 80' -c 40
curl -s http://example.com > /dev/null

E. Save a capture for Wireshark:

sudo tcpdump -i any -n -s0 -w /tmp/capture.pcap 'host example.com'
# ... generate traffic ...
# Ctrl-C, then:
wireshark /tmp/capture.pcap        # or open it in the GUI

F. A rotating capture for an intermittent problem (leave it running for hours):

sudo tcpdump -i any -n -s0 -C 100 -W 10 -w /tmp/ring.pcap 'port 443'
# keeps 10 files of 100 MB each, overwriting the oldest

Common confusions

"tcpdump shows nothing." Check: right interface (-i any is safest), filter syntax (needs quotes), and that traffic is actually happening.
"I see the request but not the response." Often the response goes via a different interface. Use -i any.
"Checksum incorrect" warnings. Normal — your network card computes checksums after tcpdump sees the packet. Ignore them for outgoing traffic.
"Can I capture on Wi-Fi?" Yes for your own traffic. Capturing other people's Wi-Fi traffic needs monitor mode and is usually illegal.

Check yourself

What does -n do and why should you always use it?
What do the flags [S], [S.], [P.], and [R] mean?
Why capture on both ends to diagnose packet loss?
What filter shows only new connection attempts?