Chapter 35 — The five classic problems and their fingerprints

A reference table. Come back to this whenever something breaks.

1. DNS failure

   SYMPTOMS:  ping <ip> works;  ping <name> fails
              "could not resolve host" / NXDOMAIN
              Works for some people, not others
              Worked yesterday, broke after a change
   CHECK:     dig <name>
              dig @1.1.1.1 <name>       (compare -- different answers = caching/propagation)
              cat /etc/resolv.conf
   CAUSES:    dead DNS server, wrong resolver, TTL not expired after a change,
              split-horizon/VPN, typo in a record

2. MTU / fragmentation black hole

   SYMPTOMS:  Connects, then HANGS
              SSH logs in then freezes
              Small pages load, big ones don't
              git clone stalls at a few KB
              Happens over VPN but not without
   CHECK:     ping -M do -s 1472 <host>       (fails? MTU is under 1500)
              tracepath <host>
   CAUSES:    VPN/tunnel overhead + a firewall blocking ICMP type 3
   FIX:       allow ICMP type 3; MSS clamping; lower the MTU

3. Firewall dropping (vs rejecting)

   SYMPTOMS:  Connection TIMES OUT (long wait, no error)
              vs "connection refused" (instant)
   CHECK:     nc -zv host port
              sudo tcpdump -n 'host X and port Y'   -- do you see SYNs going out
                                                       and nothing coming back?
   CAUSES:    firewall rule, security group, missing port forward,
              service bound to 127.0.0.1 only
   KEY:       TIMEOUT = silently dropped.  REFUSED = reached the host,
              nothing listening.

4. Packet loss

   SYMPTOMS:  Slow transfers despite a fast link
              Video calls stutter; voice breaks up
              Retransmissions in Wireshark
   CHECK:     mtr -rwzbc 100 <host>
              -> loss at hop N that CONTINUES to the destination = real
              -> loss at hop N that disappears after = cosmetic (ICMP rate limit)
              ip -s link          (interface errors?)
              ethtool -S eth0 | grep -i err
   CAUSES:    bad cable/connector, Wi-Fi interference, congested link,
              duplex mismatch, failing hardware

5. Latency / bufferbloat

   SYMPTOMS:  Everything is fine until someone downloads something
              Video calls break during backups
              Ping jumps from 20 ms to 500+ ms under load
   CHECK:     ping <somewhere> while running a big transfer
              waveform.com/tools/bufferbloat
   CAUSES:    a big dumb buffer at the bottleneck (usually your router/modem)
   FIX:       enable SQM / fq_codel / cake on the router

The bonus one: it's the application

   SYMPTOMS:  Network metrics are all fine.  No loss, low latency,
              handshakes complete, TLS fine.  But it's still slow.
   CHECK:     curl -w '... ttfb %{time_starttransfer} total %{time_total}'
              -> a big gap between connect and first byte = the SERVER is thinking
              Wireshark: a long gap with no missing packets = application delay
              ss -tin: ZeroWindow = the RECEIVER's app is too slow
   REMEMBER:  "The network is slow" is usually a database query.

The quick reference table