Appendix A — Real Incidents, Decoded

Postmortems are the best system design teachers: they show the corner cases that actually happen. For each incident below there's the cascade, the concept from this guide, and the design lesson. Interviewers at senior levels often ask "tell me about an outage and what the architecture should have done differently", so citing these, and your own, is a strong signal.

A.1 AWS S3, us-east-1, February 28, 2017: a typo removes the index

 Engineer runs playbook command to remove a few billing servers
   → input typo → removes far larger set incl. S3 INDEX and PLACEMENT subsystems
   → those subsystems need FULL RESTART (not done at this scale for years; slower than expected)
   → S3 GET/PUT/LIST fail in us-east-1 for ~4 h
   → thousands of services (and AWS's own status dashboard) that depend on S3 break

Concepts: failure domains (§14.3), blast radius and cells (§20.4), monitoring outside the failure domain (§16.7). Lessons: tooling must limit the rate and scope of capacity removal (AWS added safeguards so capacity can't drop below a minimum); partition big subsystems into cells so a restart is small and routinely exercised; don't host your status page on the thing it reports on.

A.2 GitHub, October 21, 2018: 43 seconds of partition, 24 hours of degradation

 Network maintenance → 43 s loss of connectivity between US-East hub and US-East primary DC
   → Orchestrator (MySQL HA) sees primaries unreachable → promotes US-West replicas
   → connectivity returns: writes landed in BOTH East (briefly) and West
   → apps in East now write cross-country (high latency) → degraded service
   → GitHub chooses integrity: reconcile writes, restore from backups, replay → ~24 h degraded

Concepts: failover and split brain (§8.4–8.6), consistency vs availability (§10.3), topology-aware failover (§20). Lessons: failover automation needs topology and latency awareness and guards against cross-region promotion for short blips. Know your real RPO. Sometimes the safest action is not failing over.

A.3 Cloudflare, July 2, 2019: one regex, global CPU exhaustion

 New WAF rule deployed GLOBALLY at once (rules shipped fast to respond to threats)
   → regex with catastrophic backtracking → 100% CPU on every edge server handling HTTP
   → ~27 min of 502s worldwide

Concepts: global config as the biggest failure domain (§14.3), staged rollout (§29.4), cells and rings (§20.6). Lessons: config is code, so stage it like code even when speed matters. Use regex engines with linear-time guarantees (RE2-style) for untrusted patterns. Kill switches must not depend on the broken path.

A.4 Facebook, October 4, 2021: the network removed itself from the internet

 Routine backbone maintenance command (capacity assessment) unintentionally takes down
 all backbone connections between data centers
   → DNS servers, designed to withdraw their BGP routes when they can't reach the DCs
     (a "healthy" behaviour in isolation), withdraw them → facebook.com unresolvable worldwide
   → internal tools, auth and remote access ALSO depend on the same network/DNS
   → engineers need physical access to data centers to restore → ~6 h outage

Concepts: static stability and hidden dependencies (§20.8), health checks with global effects (§4.7), out-of-band access. Lessons: a health-check reaction that's safe locally can be catastrophic when it happens everywhere at once. Break-glass access must not depend on production. Audit tools that can change everything at once.

A.5 AWS Kinesis, us-east-1, November 25, 2020: the thread limit

 Adding capacity to Kinesis front-end fleet
   → each front-end server keeps a thread per OTHER front-end server (shard-map gossip)
   → fleet size × threads exceeds OS max threads → servers fail to build shard maps
   → Kinesis down; dependent services (Cognito, CloudWatch, EventBridge, ...) impaired for hours

Concepts: Universal Scalability Law's coherency term (§22.6), N² designs, cells (§20.4). Lessons: all-to-all designs have a hidden size ceiling. Know the maximum tested cell size and scale by adding cells, not by growing one fleet. AWS moved to larger servers and a cellularized front end.

A.6 AWS DynamoDB, September 20, 2015: metastable retry storm

See §22.7. Concept: metastability (§22.3). Lesson: a dependency's response time must be budgeted with future growth of payload size (the membership data grew with GSIs). Retries without budgets turn a blip into a sustained outage.

A.7 Roblox, October 2021: 73 hours

 Consul (service discovery + KV, used by nearly everything) degrades
   → contributing factors: a newly enabled Consul streaming feature under very high load,
     and a BoltDB freelist pathology making writes slow
   → nearly all Roblox services depend on Consul → platform-wide outage
   → recovery slowed because monitoring and tooling also depended on the same systems

Concepts: coordination services on the critical path (§19.5), shared global dependencies (§14.3). Lessons: one control-plane cluster for everything is one big blast radius. Split coordination systems per domain or cell, roll out new features of critical infrastructure gradually, and keep recovery tooling independent.

A.8 Knight Capital, August 1, 2012: deploy drift

See §29.6. Concepts: deployment verification, feature-flag hygiene (§29.4, §29.7). Lesson: a deploy is a distributed system too. Verify convergence (all hosts on the same version) before enabling the flag.

A.9 CrowdStrike, July 19, 2024: a content update ships everywhere at once

 Sensor configuration/content update ("channel file") pushed broadly
   → a mismatch in the update triggers an out-of-bounds read in kernel-mode sensor code
   → ~8.5 million Windows machines crash and boot-loop
   → many need manual, physical remediation (airlines, hospitals, banks affected)

Concepts: staged rollout and rings (§20.7, §29.4), blast radius. Lessons: "data" updates that change behavior must go through the same canary and ring process as code. Validate inputs at the consumer, and design recovery for when the agent itself breaks the machine.

A.10 Slack, January 4, 2021: recovery that amplified the incident

See §22.7. Concepts: cold caches (§7.3), autoscaling as a load source, cloud networking limits. Lesson: load-test the first day after a long holiday: cold clients, cold caches, everyone reconnecting at 9 a.m.

A.11 Patterns across all of these