29. Evolution & Migrations Without Downtime

Most real senior engineering work is changing a running system. Interviewers at Google and Microsoft often follow up with "how would you migrate from the current system to this one?"

29.1 Strangler fig

                 ┌──────────────── Facade / router ────────────────┐
 clients ───────▶│ /orders/* → NEW service      (migrated)          │
                 │ /catalog/* → NEW service     (migrated)          │
                 │ everything else → LEGACY     (shrinking)         │
                 └──────────────────────────────────────────────────┘

Move one capability at a time, behind a routing layer you control, with the ability to route back.

29.2 Data migration playbook (the "expand → migrate → contract" dance)

1. Expand:     new store/schema exists alongside the old; nothing reads it yet
2. Dual-write  (or better: CDC from old → new, which avoids app-level dual-write bugs)
3. Backfill:   copy historical data, idempotently, throttled to protect production
4. Verify:     shadow reads — read both, compare, log diffs; fix until the diff rate ≈ 0
5. Cut reads:  gradually (1% → 10% → 50% → 100%) behind a flag, with fast rollback
6. Cut writes: new store becomes the source of truth; reverse CDC new → old for rollback safety
7. Contract:   after a bake period, remove the old path, the old schema, and reverse replication

29.3 Zero-downtime schema changes

Never rename or drop a column in one step. Add the new column → write both → backfill → read the new one → stop writing the old one → drop it. Each step is deployed separately and is backward compatible with the previous app version, because during a rolling deploy both versions run at once. Online schema-change tools (gh-ost, pt-online-schema-change) and NewSQL online DDL handle large tables.

29.4 Safe rollout toolkit

Feature flags decouple deploy from release (kill switches for every risky feature).
Canary deployments with automated analysis (compare error rate and latency against the baseline, and roll back automatically).
Shadow / dark traffic: send a copy of production requests to the new system and discard its responses.
Progressive delivery by cell / region / tenant tier: internal → free tier → paid → enterprise.

29.5 🏗️ Flow: Stripe-style four-phase online migration

 Phase 1  DUAL WRITE         writes → old (source of truth) + new; backfill old → new
          ├─ verify: compare old vs new for every write; fix bugs; re-backfill
 Phase 2  READ FROM NEW      reads switch to new (flag, gradual %); compare results in shadow
          ├─ rollback = flip read flag back (old still authoritative & complete)
 Phase 3  WRITE TO NEW ONLY  new becomes source of truth; (optional) reverse-sync new → old
          ├─ rollback window: reverse sync keeps old usable for N days
 Phase 4  REMOVE OLD         delete old code paths, tables, reverse sync

 Each phase is deployed and baked separately. Never combine two phases in one deploy.

29.6 🌍 In production

Stripe, "Online migrations at scale" (2017): the four-phase pattern above, used to migrate hundreds of millions of subscription objects without downtime. They used Scientist-style shadow comparison to verify reads, and ran Hadoop/MapReduce jobs for the backfill.
GitHub, Scientist library: runs old and new code paths side by side in production, compares the results, and returns the old result, so you get confidence in a refactor without risk.
Figma's sharding migration: "logical sharding" first, so queries were routed as if sharded while the data stayed on one database. That made the physical split a small, rehearsed step instead of a big-bang cutover.
Discord's message migration: after a slow first attempt, they wrote a dedicated Rust migrator that moved trillions of messages in about 9 days, while serving production traffic.
Knight Capital (2012): a deploy reached 7 of 8 servers. The eighth still ran old code, and a repurposed feature flag activated dead code there. The result was about $440M in losses in 45 minutes. Lessons: verify deploys reached every server, never reuse flag names, and delete dead code.

29.7 ⚠️ Corner cases & deep details

Backfill overwrites newer data: the backfill reads row X at 10:00, a user updates X at 10:01 (dual-written to new), and the backfill writes its 10:00 copy at 10:02, losing the update. Backfill with conditional writes ("only if absent" or "only if the version is older").
Dual-write divergence: the write to old succeeds and the write to new fails (or the reverse). Detect it with continuous reconciliation, and prefer CDC-based sync over app-level dual writes.
ID mismatches: the new system generates its own IDs, and every reference (URLs, foreign keys, external partners) breaks. Preserve IDs, or keep a mapping table forever.
Forward compatibility: after phase 3 writes new-format data, rolling back the code to an old version that can't read it is a second outage. Ship readers that understand both formats before any writer changes.
The long tail of old clients: mobile apps and partner integrations still call the old API. Keep a compatibility layer, measure usage, and set deprecation dates with real communication (§61, Stripe's versioning).
Feature-flag debt: thousands of stale flags create untested combinations (Knight Capital). Give flags owners and expiry dates, and add CI checks for stale flags.