20. Multi-Region, Cells & Disaster Recovery

20.1 Why go multi-region (pick your actual reason)

1. Latency for global users (only helps if their data is nearby too).
2. Availability / DR: survive a region loss.
3. Data residency laws.
4. Capacity: a region runs out of a resource type (common with GPUs).

Each reason leads to a different design. Ask which one applies.

20.2 Topologies

RPO (Recovery Point Objective) is how much data you can lose. RTO (Recovery Time Objective) is how long you can be down. Ask the business for both, because they map directly onto cost.

20.3 The home-region pattern (the usual Staff+ answer)

           Global routing (GeoDNS / Anycast + user→region directory)
             │                     │                      │
         ┌───▼───┐             ┌───▼───┐              ┌───▼───┐
         │US-East│◀──async────▶│EU-West│◀────async───▶│AP-SE  │
         │ cells │  replication│ cells │  replication │ cells │
         └───────┘ (for DR /   └───────┘              └───────┘
                    global reads)
Each user/tenant has a HOME region: all their writes go there (single writer → no conflicts).
Reads of other users' data can be served from local replicas (eventually consistent).
Failover: move a region's users' home to another region after the replication catch-up point.

This gives local latency for most operations, no write conflicts, a natural fit for data residency (pin EU tenants to the EU), and a manageable blast radius.

20.4 Cell-based architecture

A cell is a complete, independent copy of the service stack (app + data) serving a subset of customers. A thin routing layer maps customer → cell.

            ┌──────────── Cell router (thin, highly available, mostly static) ───────────┐
            ▼                         ▼                        ▼                         ▼
       ┌─────────┐               ┌─────────┐              ┌─────────┐               ┌─────────┐
       │ Cell 1  │               │ Cell 2  │              │ Cell 3  │     ...       │ Cell N  │
       │app+db+q │               │app+db+q │              │app+db+q │               │app+db+q │
       └─────────┘               └─────────┘              └─────────┘               └─────────┘

Benefits: blast radius = 1/N of customers; deploys roll cell by cell (cell 1 is the canary); the maximum cell size is a tested, known quantity, so scaling means adding cells instead of discovering new limits; and noisy tenants are isolated. AWS has written extensively about this, along with shuffle sharding: each customer is assigned a random combination of nodes, so two customers rarely share all of their nodes and one bad customer can't take down everyone.

Costs: cross-cell features (global search, analytics) need a separate aggregation plane; migrating tenants between cells is a project in itself; and the router must be extremely simple and reliable.

20.5 DR is a process, not an architecture

• Test failover regularly (game days, region evacuation drills). An untested DR plan is a hope, not a plan.
• Static stability: the system keeps working in failover without needing to make control-plane calls (e.g. pre-provisioned capacity rather than "autoscale in the surviving region during the disaster").
• Watch out for hidden single-region dependencies: identity, secrets, DNS control plane, CI/CD, container registry, feature-flag service.

20.6 🏗️ Flows: region evacuation and cell routing

Region evacuation (planned or emergency):

 1. Detect: SLO burn in region EU-1 (or a planned drill)
 2. Decide: automated for stateless tiers; human-approved if data loss (RPO) is possible
 3. Pre-check: target regions have headroom (static stability: capacity pre-provisioned)
 4. Shift traffic: global LB weights EU-1 100% → 0% in steps (10% / min) to avoid cold-cache stampede
 5. Data: for home-region users of EU-1 → promote replicas elsewhere after replication catch-up point
 6. Verify: SLOs in receiving regions, error budgets, queue backlogs draining
 7. Failback later: reverse replication, re-sync, shift back gradually (failback is its own risky change)

Cell router:

 request ─▶ edge ─▶ cell router: cell = directory[tenant_id] (cached map, versioned, tiny)
                          │ unknown tenant → placement service assigns least-loaded cell
                          ▼
                     Cell 7 (complete stack) ── only Cell 7's tenants affected by its failure
 Deploys: cell 1 (canary, internal tenants) → wait/bake → 1% → 10% → 50% → all cells

20.7 🌍 In production

• Netflix active-active (2013 onward): serves from multiple AWS regions, with Cassandra replicating data across regions and Zuul steering traffic. Netflix regularly practices region evacuation, moving all of a region's traffic to the others, so a real regional failure is a routine operation.
• AWS cell-based architecture and shuffle sharding: AWS describes cells as a core blast-radius technique (Builders' Library, re:Invent talks). Route 53 uses shuffle sharding so each customer's DNS zone is served by a unique combination of name servers, and a DDoS against one customer rarely takes down another's whole set.
• Slack's move to a cellular architecture (2023): after AZ-level gray failures, Slack restructured so traffic can be drained from an AZ quickly, with each AZ acting as a cell for most services.
• Shopify pods: each pod holds a set of shops with their own MySQL, Redis, and Memcached. Pods can be moved between data centers or regions (pod mobility), which also supports disaster recovery.
• Azure safe deployment practices: changes roll out through rings (internal canary → pilot → light region → broad), with paired regions never updated at the same time. It's the same blast-radius idea applied to deploys.

20.8 ⚠️ Corner cases & deep details

• Failover capacity: with 3 regions at 70% utilization, losing one puts 105% on the other two, so the failover itself causes an outage. Run each region at ≤ (N−1)/N × safe utilization (e.g. ≤ ~45% for 3 regions at a 70% ceiling).
• Cold caches on failover: the receiving region's caches don't hold the evacuated users' data, so the DB takes a miss storm. Shift traffic gradually, and pre-warm or replicate hot caches.
• Gray failures: the region isn't down, just 5% slower or erroring. Health checks pass, so automated failover never triggers. Detect from client-side SLIs, and give humans a big "evacuate" button.
• Global control-plane dependencies: the data plane is multi-region, but IAM, DNS changes, certificate issuance, or deploy tooling run in one region. When that region fails you can't execute the failover. Audit dependencies for static stability.
• Residency vs DR: EU data must stay in the EU, so the DR region must also be in the EU. Plan region pairs within each legal boundary.
• Users who span cells: shared channels between companies in different cells, or a marketplace between buyer and seller cells. You need a cross-cell service or an explicit "home cell owns the shared object" rule.
• The cell router is a global dependency: keep it trivially simple, statically stable (cached map, survives control-plane loss), and deployed more carefully than anything else.