Chapter 55 — Incident response for cloud and containers

How it works

"Isolate, don't terminate" — the central adjustment for ephemeral compute:

   The instinct to "kill it" is usually WRONG in cloud/container IR,
   because it destroys evidence AND doesn't stop a credential's blast
   radius (below). The pattern:
     CLOUD INSTANCE: don't terminate. Instead: detach it from the load
       balancer/auto-scaling group (so it stops serving traffic and
       won't be auto-replaced mid-investigation), snapshot the EBS
       volume(s) WITHOUT stopping the instance where your platform
       supports it, attempt memory acquisition if feasible (a LiME
       capture on Linux, or a hypervisor-level memory snapshot if your
       platform offers one), tag it clearly as "under investigation" so
       automation doesn't reap it, THEN isolate its network access
       (a quarantine security group allowing only your forensics
       tooling).
     KUBERNETES POD: `kubectl cordon` the NODE (stops new pods scheduling
       there, doesn't touch the running one). Do NOT delete the pod.
       Apply a QUARANTINE NetworkPolicy (deny all except a forensics
       egress) via a label selector rather than deleting/recreating.
       Extract logs and, where your CRI/kernel supports checkpointing
       (CRIU-based), a CONTAINER CHECKPOINT before any termination.
       `kubectl debug` can attach an ephemeral debug container sharing
       the target pod's process namespace for LIVE triage without
       modifying the compromised container itself.
     ONLY AFTER evidence capture: terminate/delete and REBUILD from a
       known-good image (Part 3's paved-road pipeline) rather than
       "cleaning" the compromised instance/container in place -- you
       cannot be confident you found every persistence mechanism by
       inspection alone (G1 Ch 51's eradication principle, doubly true
       here where rebuild is cheap and fast).

Credential blast-radius containment — the cloud-specific version of "reset the password":

   A traditional IR reflex ("reset the compromised account's password")
   doesn't map cleanly to cloud identity, because:
     - an ACTIVE STS SESSION (Part 2 Ch 5) keeps working until it expires
       on its own -- there is no direct "kill this session" API in most
       clouds. The practical response: attach an EXPLICIT DENY inline
       policy to the underlying role/user IMMEDIATELY (Part 2 Ch 6's
       policy-evaluation rule: an explicit deny wins over everything,
       instantly, for all NEW calls even mid-session) while you
       investigate; this doesn't retroactively undo anything already
       done, but it stops FURTHER action within seconds, faster than
       waiting for a full credential rotation to propagate.
     - a WORKLOAD IDENTITY (IRSA/Workload Identity/SPIFFE SVID, Part 4
       Ch 20 / Part 5 Ch 23) is short-lived by design (a genuine security
       win from those chapters) -- but if the underlying TRUST (the OIDC
       provider config, the SPIRE registration entry) is what's
       compromised rather than one session, you must revoke/rotate the
       TRUST relationship itself, not just wait for one token to expire.
     - a SINGLE SSRF or RCE can expose a role usable against DOZENS of
       resources (Part 2's attack-path chapters) -- credential
       containment in cloud IR is therefore often BROADER in blast-radius
       terms than a traditional single-account compromise, and the
       explicit-deny-first, investigate-second pattern exists specifically
       because full, careful remediation takes longer than the exposure
       window can safely tolerate.
   AFTER containment: fully rotate/replace the underlying credential
   material, review and fix the trust/permission misconfiguration that
   allowed the exposure (Part 2 Ch 6, Part 3 Ch 12), and only then remove
   the emergency deny policy.

Cloud-native forensic evidence sources (recap and prioritise):

   PRIMARY TIMELINE: CloudTrail / Cloud Audit Logs / Kubernetes audit
     logs (Ch 52) -- usually your fastest, richest source, IF retention
     was adequate (Ch 52's dwell-time argument, made concrete: this is
     exactly the evidence that ages out if retention is too short).
   ENCRYPTION CONTEXT on KMS Decrypt calls (Part 7 Ch 44's worked
     example) -- lets you scope EXACTLY which records were accessed
     during a compromise window, not just "the key was used."
   RUNTIME TELEMETRY: Falco/Tetragon event history (Part 4 Ch 20) for
     container-level process/syscall evidence.
   VPC FLOW LOGS: network-level corroboration of what a compromised
     workload actually connected to, cross-referenced against the
     control-plane timeline.
   IMMUTABLE SNAPSHOTS: EBS/disk snapshots taken during isolation
     (above), hashed and stored with chain-of-custody records exactly as
     G1 Ch 51 describes, just applied to cloud storage objects instead of
     physical drives.

IR automation / SOAR — necessary because cloud/container compromise moves faster than a human on-call can react to manually:

   Security Orchestration, Automation and Response platforms (Tines, Torq,
   Palo Alto XSOAR) or cloud-native automation (a high-confidence
   GuardDuty/Security Hub finding triggering a Lambda/Cloud Function that
   automatically attaches the quarantine policy, cordons the node, or
   snapshots the volume) let CONTAINMENT happen in seconds instead of
   whatever your on-call's pager-to-keyboard latency is.
   THE TRADE-OFF: gate fully-automated, DESTRUCTIVE or DISRUPTIVE actions
   (isolating a production credential, cordoning a node serving live
   traffic) behind HIGH-CONFIDENCE detections only (Ch 51's precision
   metric matters enormously here -- a false positive that auto-isolates
   a production payment role is itself an incident). For ambiguous or
   lower-confidence findings, automate the INVESTIGATION step (gather
   evidence, open a ticket, page someone) but keep a human in the loop
   for the containment DECISION.

Cross-account/cross-cluster coordination: Part 2 Ch 7's multi-account structure puts your IR tooling and centralised, tamper-resistant logs in the dedicated security account — investigations that span multiple workload accounts or clusters are coordinated FROM there, which is exactly why that account's own access (Ch 49) must be the most tightly governed of all.