Frontend Labs
⋯

Chapter 51 — Incident response

In one sentence

Incident response is a rehearsed process — prepare, detect, contain, eradicate, recover, learn — that turns a chaotic "we're breached" moment into a series of known steps, and the preparation matters more than the heroics.

The lifecycle (NIST SP 800-61)

   1. PREPARATION      before anything happens
      - IR plan + playbooks (ransomware, cred compromise, data exfil,
        web-app breach, insider, supply-chain, DDoS)
      - roles: Incident Commander, Comms lead, Scribe, technical leads,
        Legal, Exec sponsor. A RACI. On-call rotation.
      - contact tree (incl. out-of-band: what if Slack/email is
        compromised?), retainer with an IR/forensics firm, cyber-insurance
        contacts, law-enforcement contacts
      - tooling ready: EDR, centralised logs (Ch 50), ability to isolate a
        host/rotate all creds/pull an image, a clean-room comms channel
      - RUNBOOKS TESTED via tabletop exercises + at least one live drill/yr

   2. DETECTION & ANALYSIS
      - triage: is it real? scope? severity? (define severities in advance)
      - declare an incident; assign an IC; start the timeline doc
        (every action, UTC, who) -- this is evidence and post-mortem input
      - preserve evidence BEFORE you change things (see forensics below)

   3. CONTAINMENT
      - SHORT-TERM: stop the bleeding -- isolate hosts (network quarantine,
        not power-off if you want memory), disable compromised accounts,
        revoke tokens/keys, block C2 IPs/domains, take the app to
        maintenance mode if needed
      - decide: monitor-and-learn vs contain-now (tips toward contain-now
        if data is actively leaving)
      - LONG-TERM: temporary fixes that let you operate while eradicating
        (patched hosts rebuilt from known-good, tighter rules)

   4. ERADICATION
      - remove the foothold: rebuild (don't "clean") compromised systems
        from trusted images, patch the entry vector, reset ALL potentially
        exposed credentials, remove persistence (cron, services, keys,
        webshells, IAM backdoors, OAuth grants)
      - confirm the initial access vector is actually closed

   5. RECOVERY
      - restore from known-good backups (verify integrity, scan first),
        return systems to prod in a monitored, staged way
      - heightened monitoring for recurrence (attackers retry)
      - define exit criteria: when is the incident "closed"?

   6. POST-INCIDENT ("lessons learned")
      - BLAMELESS post-mortem within ~2 weeks: timeline, root cause (5
        whys / causal analysis), what worked, what didn't, dwell time,
        MTTD/MTTC
      - concrete action items with owners and dates; track to done
      - update playbooks, detections, and controls

Containment decisions under pressure

   Pull network, keep power    -> preserves RAM (malware, keys, live
                                  processes) for forensics. Preferred if
                                  you have the capability.
   Power off                   -> stops encryption/exfil fast, but destroys
                                  volatile evidence. Sometimes the right
                                  call (active ransomware encryption).
   Leave running, watch        -> only if exfil is NOT ongoing and you have
                                  the monitoring to learn scope safely.
   Rotate credentials WIDELY   -> almost always, early. Assume the blast
                                  radius is bigger than it looks.

Forensic basics (do no harm to the evidence)

   ORDER OF VOLATILITY (collect most-volatile first):
     CPU registers/cache -> RAM -> network state (connections, ARP, routes)
     -> running processes -> disk -> logs/archives -> physical config

   PRINCIPLES:
     - work on COPIES; hash originals (SHA-256) and record the hash
     - maintain CHAIN OF CUSTODY (who touched what, when, why)
     - capture memory before shutdown (avml, LiME, WinPMEM)
     - image disks with a write-blocker; `dd`/` dc3dd` + hash
     - note system time vs true time (clock skew)
     - if it may go legal, involve Legal EARLY and follow their guidance
   You don't need to be a forensic examiner -- you need to not TRAMPLE the
   scene before one arrives.

Comms and obligations

   INTERNAL: single source of truth (the timeline doc + a status channel),
     regular exec updates, need-to-know for details.
   EXTERNAL: legal/regulatory timelines are real and short:
     - GDPR: notify the supervisory authority within 72 HOURS of becoming
       aware of a personal-data breach (and affected individuals if high
       risk).
     - US: state breach laws, SEC 4-day material-incident disclosure for
       public companies, sector rules (HIPAA, PCI, GLBA), CIRCIA.
     - contractual: customer notification SLAs.
   Prepare holding statements in advance. Do NOT speculate publicly on
   cause/scope before you know. Coordinate with Legal + PR.

Practice (40 min) — tabletop

   Run a 45-minute tabletop with 2-4 people (or solo, writing it out).

   SCENARIO: "At 02:14 an alert fires: SecureShop's orders-api tier made
   200 outbound HTTPS connections to an IP in a country you don't operate
   in. A dev notices the admin panel has a new user 'svc_backup' created
   at 01:50. crt.sh shows a cert issued for admin.securesh.op by a CA you
   don't use, 3 days ago."

   Work through, writing decisions + timestamps:
   1. Is this an incident? Severity? Who is IC? What's the first action in
      the next 5 minutes?
   2. Contain: what do you isolate/disable/revoke, in what order? What do
      you preserve first?
   3. What logs/data do you pull, and from where (Ch 50)? What's the
      likely initial vector given the cert + new user + egress?
   4. Eradicate: rebuild vs clean? Which credentials get rotated? How do
      you know the door is shut?
   5. Recover: backup strategy, exit criteria, extra monitoring.
   6. Obligations: is personal data implicated? Clock started when? Who do
      you notify and by when?
   7. Post-mortem: 3 action items you'd bet money on.

   Write it to ~/sec-lab/reports/part11-ir-tabletop.md.

Common confusions

  • "IR is what you do after a breach." 80% of IR is preparation — plans, playbooks, tooling, drills. Mid-incident is a bad time to discover you can't isolate a host or find the logs.
  • "Wipe and reinstall the infected box." That destroys evidence and you may miss the entry vector and persistence, guaranteeing a repeat. Preserve, then rebuild from known-good.
  • "Change the one password we know was phished." Assume broader compromise; rotate widely and early.
  • "Post-mortems find who to blame." Blameless post-mortems find systemic causes. Blame makes people hide information in the next incident.
  • "We'll notify once we fully understand it." Regulatory clocks (GDPR 72h, SEC 4 days) start at awareness, not at full understanding. Involve Legal immediately.

Check yourself

  1. Name the six phases of the IR lifecycle. Which consumes most of the effort?
  2. Why prefer "pull network, keep power" over powering off — and when is powering off right?
  3. State the order of volatility for evidence collection.
  4. What is chain of custody and why does it matter?
  5. When does the GDPR 72-hour clock start?
  6. What makes a post-mortem "blameless" and why does that matter?

Further reading

  • Standard: NIST SP 800-61 Rev. 2 (Computer Security Incident Handling Guide) — the canonical lifecycle. SANS "Incident Handler's Handbook" for a shorter version.
  • Book: Incident Response & Computer Forensics (3rd ed.), Luttgens et al.; Intelligence-Driven Incident Response, Scott Roberts.
  • Playbooks: the PagerDuty Incident Response docs (response.pagerduty.com), the Google SRE "Managing Incidents" chapter, and public IR playbook repos (Counteractive, AWS IR runbooks).
  • Practice: tabletop scenario packs (CISA Tabletop Exercise Packages); thehive-project.org (case management); Atomic Red Team for realistic telemetry.
  • Legal: your jurisdiction's breach-notification summary (IAPP maintains good charts); GDPR Art. 33/34 text.

Generated from real-life-guide.md · 8 Oct 2026 16:36