8. Scaling Basics: Replication, Read Replicas, Statelessness

8.1 Stateless services

Move all state (sessions, uploads in progress, caches that must be consistent) out of the app tier and into stores built for it. Then any instance can serve any request, autoscaling is trivial, and deploys are rolling restarts.

Things that are secretly stateful: local file uploads, in-memory rate-limit counters, WebSocket connections, scheduled cron jobs running on "every instance", and local caches that must be invalidated.

8.2 Leader–follower replication and replication lag

            writes
 clients ─────────▶ Primary ──async WAL stream──▶ Replica 1  ◀── reads
                       │                         Replica 2  ◀── reads
                       └─sync (optional)───────▶ Standby (for failover)

• Async replication: fast writes, but replicas lag (milliseconds normally, minutes under load), and failover can lose committed writes.
• Sync replication (to at least one standby): no data loss on failover, higher write latency, and writes block if the standby is down (semi-sync degrades to async).

Read-your-writes problem: a user updates their profile, the page reloads from a lagging replica, and they see the old value. Fixes:

1. Route a user's reads to the primary for N seconds after they write (sticky flag in session).
2. Track the write's LSN/GTID and read from a replica only once it has caught up to it.
3. Read from the primary for "own" data and from replicas for others' data.

8.3 Vertical vs horizontal scaling

Vertical scaling buys more time than people assume: a single cloud instance can have hundreds of vCPUs and terabytes of RAM. "Scale up the DB, scale out the app" is a legitimate answer for years of growth.

8.4 🏗️ Flow: what actually happens during a primary failover

 t0   Primary P dies (or is unreachable from the monitor ← not the same thing!)
 t0+  Failure detector (Orchestrator / Patroni / RDS) misses N health checks (≈10-30 s)
 t1   Choose candidate: replica with most recent position (least lag)
 t2   FENCE old primary: STONITH / revoke its VIP / set read_only / remove from DNS
      ── skip this and you risk split-brain if P was only partitioned, not dead
 t3   Promote replica R → new primary; repoint other replicas to R
 t4   Update routing: proxy config / DNS / service discovery → R
 t5   Clients: connection pools still hold sockets to P → errors until they reconnect
 t6   Reconcile: transactions committed on P but not replicated to R are LOST (async)
      or must be manually reconciled

8.5 🌍 In production

• GitHub, October 21, 2018: a 43-second network partition between GitHub's US East Coast network hub and its primary US East data center. Orchestrator, the MySQL failover tool, promoted primaries in the US West data center. When connectivity returned, both sides had accepted writes. Applications in the East now paid cross-country latency to the new West primaries. GitHub chose data integrity over a fast recovery, and the service was degraded for about 24 hours while they reconciled. Lessons: failover automation must be aware of topology and latency, a short partition can do far more damage than a dead server, and sometimes not failing over is the better choice.
• GitHub, 2021 ("Partitioning GitHub's relational databases to handle scale"): they grouped tables into schema domains, banned cross-domain joins and transactions using linters, and then moved domains onto separate clusters. This is how to split a monolithic database safely.
• Shopify runs MySQL with automated failover inside each pod, which limits the blast radius of a failover to that pod's shops.

8.6 ⚠️ Corner cases & deep details