Chapter 60 — Incident response: runbooks, packet evidence, and escalation

In one sentence

During an outage, your job is to narrow the blast radius, preserve evidence, communicate clearly, and change one thing at a time. After it, your job is to make the same failure easier to detect and harder to repeat.

The problem

Network incidents are unusually prone to chaos. Symptoms appear far from the cause ("the app is down" when a route changed), several teams own pieces of the path, evidence disappears when someone restarts something, and everyone has a theory. A simple, practised structure beats individual brilliance.

Roles, even for small incidents

Role	Job	Not their job
Incident commander (IC)	Owns the incident: decides priorities, assigns work, decides when to mitigate	Debugging. The IC coordinates and doesn't type commands
Operations lead(s)	Investigate and change systems, one change at a time, announced first	Talking to customers
Communications lead	Status updates to stakeholders and customers on a fixed cadence (e.g. every 30 min)	Speculating on root cause
Scribe (often the IC early on)	A timestamped log of observations, hypotheses, and actions	Filtering what seems unimportant

On a two-person team, one person is IC and comms and the other operates. The point is that someone is deciding and someone is writing it down.

First 10 minutes

1. What changed?          deploys, config, firewall, DNS, certificates, provider status pages
2. Who is affected?       one user, one AZ, one region, one ISP, everyone? since when?
3. Which layer?           DNS, routing, firewall, TLS, load balancer, app, dependency?
4. Is there a safe rollback?   rolling back a recent change beats debugging it live
5. What evidence do we have before changing state?

Scope is the fastest discriminator. "Only users on one ISP" points at routing or peering. "Only one AZ" points at infrastructure. "Everyone, since 09:05" points at whatever changed at 09:05.

Evidence bundle

Collect before and after any change, with UTC timestamps:

date -u
ip addr; ip route; ip rule
ss -s; ss -tin | head -50
nstat -az | grep -Ei 'retrans|overflow|drop|reset'
dig +short service.example.com; dig +short service.example.com @1.1.1.1
curl -v --max-time 10 -w '\ndns=%{time_namelookup} tcp=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer}\n' \
     -o /dev/null https://service.example.com/healthz
sudo tcpdump -ni any -s 0 -w incident-$(date -u +%H%M).pcap 'host 203.0.113.10'

For intermittent problems, leave a ring-buffer capture running so the evidence exists when the problem next happens:

sudo tcpdump -ni eth0 -s 128 -C 100 -W 20 -w /var/tmp/ring.pcap 'tcp port 443'
# 20 files × 100 MB, oldest overwritten; -s 128 keeps headers, not payloads

Captures can contain credentials and personal data, so treat them as sensitive and delete them when the investigation ends.

Hypothesis discipline

Write every hypothesis in the incident log in this form:

09:24  H3: backend subnet NACL drops return traffic since rule change CR-1234
       test: tcpdump on LB shows SYN-ACK from backend? on backend shows SYN arriving?
       result 09:29: SYN arrives at backend, SYN-ACK leaves backend, never reaches LB -> SUPPORTED

One hypothesis, one test, one result. It stops the team from changing three things at once and losing track of what actually fixed it.

Escalation template

Impact: 30% of users in us-east cannot reach api.example.com
Started: 2026-09-19 09:12 UTC
Evidence: SYN reaches load balancer; no SYN-ACK from backend subnet
Not DNS: authoritative and public resolvers return expected IP
Not client-side: reproduced from two networks
Recent changes: firewall rule update at 09:05 UTC
Rollback: revert rule change CR-1234
Need: network/firewall owner to confirm drop counters

Good escalation is specific. "Network is broken" is not actionable. "SYN arrives, SYN-ACK leaves, never reaches the LB, since 09:05" is.

After the incident: the blameless postmortem

Summary          one paragraph, written for someone outside the team
Impact           who, how many, how long, which SLOs were burned
Timeline (UTC)   detection, escalation, mitigation, resolution, with evidence links
Root cause       the mechanism, not a person ("a NACL rule lacked the ephemeral return range")
Contributing     why it wasn't caught: no policy tests, no alert on SYN-ACK drops, ...
What went well   keep doing these
Action items     each with an owner and a date: prevent, detect sooner, mitigate faster

"Blameless" doesn't mean nobody is accountable. It means the system that let a reasonable person make the mistake is what gets fixed. For network incidents, the best action items are usually tests (Chapter 53's policytest), alerts (Chapter 47), and runbooks that would have cut the time to diagnosis.

Common confusions

"Root-cause first, then fix." Mitigate first (roll back, fail over, shed load), then find the root cause, using evidence you captured before mitigating.
"The status page can wait." Customers notice silence. Regular updates, even "still investigating", reduce support load and build trust.
"Restarting fixed it, so we're done." A restart is a mitigation. Without evidence, the incident will happen again (Linux guide, Chapter 72).

Check yourself

Name the four incident roles and the one thing each should not do.
Why capture evidence before rolling back?
Rewrite "the database is unreachable" as a specific escalation.
What makes an action item good?