Part 14 — Debugging by layer: the incident playbook
14.1 Three strategies
| Strategy | Start at | Use when |
|---|---|---|
| Bottom-up | L1 (link, cables, errors) and work upward | "Nothing works"; new network/hardware; one host is affected |
| Top-down | L7 (the error message, logs) and work downward | One app/endpoint is broken while others work |
| Divide and conquer | L3/L4 (ping, nc) |
Most of the time: one test splits the stack in half |
14.2 The ladder: one command per layer
Run these from the failing client, in order. The first one that fails tells you the layer.
# L1/L2: do I have a link and an address?
ifconfig en0 | grep -E 'status|inet ' # Linux: ip -br addr
# L2/L3: can I reach my gateway?
ping -c 3 $(route -n get default | awk '/gateway/{print $2}') # Linux: ip route | awk '/default/{print $3}'
# L3: can I reach the Internet by IP (no DNS)?
ping -c 3 1.1.1.1
# L7 (DNS): can I resolve the name?
dig +short shop.example.com
# L3: can I reach the target IP? (ICMP may be blocked; don't trust "fail" alone)
ping -c 3 198.51.100.20 ; mtr -rwc 20 198.51.100.20
# L4: is the port open?
nc -vz shop.example.com 443
# L5/L6: does TLS negotiate, with a valid certificate?
openssl s_client -connect shop.example.com:443 -servername shop.example.com </dev/null
# L7: does the app answer correctly?
curl -sv https://shop.example.com/api/v1/health
14.3 One curl command that splits latency by layer
This is the most useful single command in this guide:
curl -o /dev/null -s -w '
DNS lookup (L7 DNS) : %{time_namelookup}s
TCP connect (L4) : %{time_connect}s
TLS done (L5/L6) : %{time_appconnect}s
Request sent : %{time_pretransfer}s
First byte (server work) : %{time_starttransfer}s
Total : %{time_total}s
HTTP %{http_code}, %{size_download} bytes, remote %{remote_ip}, HTTP/%{http_version}
' https://shop.example.com/api/v1/health
The values are cumulative. Subtract consecutive values to get each phase:
| Large gap between… | Means | Look at |
|---|---|---|
0 → namelookup |
Slow DNS | Resolver, TTLs, DNS provider |
namelookup → connect |
High RTT or SYN loss/retries (a gap ≈1 s or ≈3 s = retransmitted SYN) | Network path, firewall, backlog |
connect → appconnect |
Slow TLS (OCSP, large cert chain, CPU-starved TLS terminator, extra RTT) | LB/TLS config |
pretransfer → starttransfer |
Server think time (the app or its dependencies) | APM traces, DB |
starttransfer → total |
Large body / low throughput / loss | Payload size, compression, BDP |
14.4 Symptom → layer quick map
| What the user/engineer says | Most likely layer(s) | First check |
|---|---|---|
| "No internet at all" | L1–L3 | Link, IP address, gateway ping |
| "Some sites work, others don't" | L3 DNS/L7, MTU | dig, PMTU ping test |
| "It's slow, but only sometimes / only p99" | L1/L2 loss → L4 retransmits | mtr loss, ss -ti retrans, interface errors |
| "Connection refused" | L4 (nothing listening) | Process up? Right port? |
| "Connection timed out" | L3/L4 filtering | Security groups, NACLs, routes |
| "Connection reset" | L4/L5 | Idle timeouts, crashes, keep-alive mismatch |
| "SSL/certificate error" | L6 | Expiry, chain, SNI, client clock |
| "502 / 504" | L4–L7 between proxy and app | LB logs vs app logs, timeouts chain |
| "500 / wrong data" | L7 (or L6 serialisation) | App logs, traces |
| "Works locally, fails in Kubernetes" | L3 (network policy), L7 (DNS/service names), L2 (MTU) | kubectl exec + the ladder from inside the pod |
14.5 Packet capture: ground truth
When logs disagree, the packets decide:
# Capture on a server, filtered to one client, full packets, to a file
sudo tcpdump -i any -nn -s0 -w /tmp/case123.pcap host 203.0.113.7 and port 443
# On Kubernetes: capture inside a pod's network namespace
kubectl debug -it pod/orders-7d9f --image=nicolaka/netshoot --target=orders -- tcpdump -nn -i any port 8080
Open it in Wireshark and use Statistics → Conversations, Analyze → Expert
Information (retransmissions, zero windows, resets), and Statistics → TCP
Stream Graphs. For TLS, export browser keys with SSLKEYLOGFILE=/tmp/keys.log
and point Wireshark at that file to see decrypted HTTP.
Go deeper in the series
- The ladder as a program:
netcheckstops at the first failing layer (TCP/IP Chapter 34). curl -win Go:httptracetimes each phase and shows connection reuse (HTTPS Chapter 17).- Decode a capture yourself:
pcapread(TCP/IP Chapter 32). - Running the incident: TCP/IP Chapter 60 and Linux Chapter 72.
- Security incidents in cloud and containers, and detections as tested Go code: Security Engineering in Depth, Chapters 51 and 55.