Chapter 51 — Incident response
In one sentence
Incident response is a rehearsed process — prepare, detect, contain, eradicate, recover, learn — that turns a chaotic "we're breached" moment into a series of known steps, and the preparation matters more than the heroics.
The lifecycle (NIST SP 800-61)
1. PREPARATION before anything happens
- IR plan + playbooks (ransomware, cred compromise, data exfil,
web-app breach, insider, supply-chain, DDoS)
- roles: Incident Commander, Comms lead, Scribe, technical leads,
Legal, Exec sponsor. A RACI. On-call rotation.
- contact tree (incl. out-of-band: what if Slack/email is
compromised?), retainer with an IR/forensics firm, cyber-insurance
contacts, law-enforcement contacts
- tooling ready: EDR, centralised logs (Ch 50), ability to isolate a
host/rotate all creds/pull an image, a clean-room comms channel
- RUNBOOKS TESTED via tabletop exercises + at least one live drill/yr
2. DETECTION & ANALYSIS
- triage: is it real? scope? severity? (define severities in advance)
- declare an incident; assign an IC; start the timeline doc
(every action, UTC, who) -- this is evidence and post-mortem input
- preserve evidence BEFORE you change things (see forensics below)
3. CONTAINMENT
- SHORT-TERM: stop the bleeding -- isolate hosts (network quarantine,
not power-off if you want memory), disable compromised accounts,
revoke tokens/keys, block C2 IPs/domains, take the app to
maintenance mode if needed
- decide: monitor-and-learn vs contain-now (tips toward contain-now
if data is actively leaving)
- LONG-TERM: temporary fixes that let you operate while eradicating
(patched hosts rebuilt from known-good, tighter rules)
4. ERADICATION
- remove the foothold: rebuild (don't "clean") compromised systems
from trusted images, patch the entry vector, reset ALL potentially
exposed credentials, remove persistence (cron, services, keys,
webshells, IAM backdoors, OAuth grants)
- confirm the initial access vector is actually closed
5. RECOVERY
- restore from known-good backups (verify integrity, scan first),
return systems to prod in a monitored, staged way
- heightened monitoring for recurrence (attackers retry)
- define exit criteria: when is the incident "closed"?
6. POST-INCIDENT ("lessons learned")
- BLAMELESS post-mortem within ~2 weeks: timeline, root cause (5
whys / causal analysis), what worked, what didn't, dwell time,
MTTD/MTTC
- concrete action items with owners and dates; track to done
- update playbooks, detections, and controls
Containment decisions under pressure
Pull network, keep power -> preserves RAM (malware, keys, live
processes) for forensics. Preferred if
you have the capability.
Power off -> stops encryption/exfil fast, but destroys
volatile evidence. Sometimes the right
call (active ransomware encryption).
Leave running, watch -> only if exfil is NOT ongoing and you have
the monitoring to learn scope safely.
Rotate credentials WIDELY -> almost always, early. Assume the blast
radius is bigger than it looks.
Forensic basics (do no harm to the evidence)
ORDER OF VOLATILITY (collect most-volatile first):
CPU registers/cache -> RAM -> network state (connections, ARP, routes)
-> running processes -> disk -> logs/archives -> physical config
PRINCIPLES:
- work on COPIES; hash originals (SHA-256) and record the hash
- maintain CHAIN OF CUSTODY (who touched what, when, why)
- capture memory before shutdown (avml, LiME, WinPMEM)
- image disks with a write-blocker; `dd`/` dc3dd` + hash
- note system time vs true time (clock skew)
- if it may go legal, involve Legal EARLY and follow their guidance
You don't need to be a forensic examiner -- you need to not TRAMPLE the
scene before one arrives.
Comms and obligations
INTERNAL: single source of truth (the timeline doc + a status channel),
regular exec updates, need-to-know for details.
EXTERNAL: legal/regulatory timelines are real and short:
- GDPR: notify the supervisory authority within 72 HOURS of becoming
aware of a personal-data breach (and affected individuals if high
risk).
- US: state breach laws, SEC 4-day material-incident disclosure for
public companies, sector rules (HIPAA, PCI, GLBA), CIRCIA.
- contractual: customer notification SLAs.
Prepare holding statements in advance. Do NOT speculate publicly on
cause/scope before you know. Coordinate with Legal + PR.
Practice (40 min) — tabletop
Run a 45-minute tabletop with 2-4 people (or solo, writing it out).
SCENARIO: "At 02:14 an alert fires: SecureShop's orders-api tier made
200 outbound HTTPS connections to an IP in a country you don't operate
in. A dev notices the admin panel has a new user 'svc_backup' created
at 01:50. crt.sh shows a cert issued for admin.securesh.op by a CA you
don't use, 3 days ago."
Work through, writing decisions + timestamps:
1. Is this an incident? Severity? Who is IC? What's the first action in
the next 5 minutes?
2. Contain: what do you isolate/disable/revoke, in what order? What do
you preserve first?
3. What logs/data do you pull, and from where (Ch 50)? What's the
likely initial vector given the cert + new user + egress?
4. Eradicate: rebuild vs clean? Which credentials get rotated? How do
you know the door is shut?
5. Recover: backup strategy, exit criteria, extra monitoring.
6. Obligations: is personal data implicated? Clock started when? Who do
you notify and by when?
7. Post-mortem: 3 action items you'd bet money on.
Write it to ~/sec-lab/reports/part11-ir-tabletop.md.
Common confusions
- "IR is what you do after a breach." 80% of IR is preparation — plans, playbooks, tooling, drills. Mid-incident is a bad time to discover you can't isolate a host or find the logs.
- "Wipe and reinstall the infected box." That destroys evidence and you may miss the entry vector and persistence, guaranteeing a repeat. Preserve, then rebuild from known-good.
- "Change the one password we know was phished." Assume broader compromise; rotate widely and early.
- "Post-mortems find who to blame." Blameless post-mortems find systemic causes. Blame makes people hide information in the next incident.
- "We'll notify once we fully understand it." Regulatory clocks (GDPR 72h, SEC 4 days) start at awareness, not at full understanding. Involve Legal immediately.
Check yourself
- Name the six phases of the IR lifecycle. Which consumes most of the effort?
- Why prefer "pull network, keep power" over powering off — and when is powering off right?
- State the order of volatility for evidence collection.
- What is chain of custody and why does it matter?
- When does the GDPR 72-hour clock start?
- What makes a post-mortem "blameless" and why does that matter?
Further reading
- Standard: NIST SP 800-61 Rev. 2 (Computer Security Incident Handling Guide) — the canonical lifecycle. SANS "Incident Handler's Handbook" for a shorter version.
- Book: Incident Response & Computer Forensics (3rd ed.), Luttgens et al.; Intelligence-Driven Incident Response, Scott Roberts.
- Playbooks: the PagerDuty Incident Response docs (
response.pagerduty.com), the Google SRE "Managing Incidents" chapter, and public IR playbook repos (Counteractive, AWS IR runbooks). - Practice: tabletop scenario packs (CISA Tabletop Exercise Packages);
thehive-project.org(case management); Atomic Red Team for realistic telemetry. - Legal: your jurisdiction's breach-notification summary (IAPP maintains good charts); GDPR Art. 33/34 text.