4. Building Blocks & Scaling from 1 to 10 Million Users

4.1 The canonical web stack

                         ┌───────────┐
   Users ──DNS──────────▶│   CDN     │── static assets, cached API responses
      │                  └─────┬─────┘
      │                        ▼
      │               ┌──────────────────┐
      └──────────────▶│ L7 Load Balancer │  TLS termination, routing, WAF
                      │  / API Gateway   │  authN, rate limiting
                      └────────┬─────────┘
                               ▼
                 ┌─────────────────────────────┐
                 │  Stateless app servers (N)  │  autoscaled
                 └──┬───────────┬──────────┬───┘
                    ▼           ▼          ▼
              ┌────────┐  ┌──────────┐ ┌───────────┐
              │ Cache  │  │ Primary  │ │  Queue /  │──▶ async workers
              │(Redis) │  │   DB     │ │  Stream   │
              └────────┘  └────┬─────┘ └───────────┘
                               │ replication
                          ┌────▼─────┐      ┌───────────────┐
                          │ Replicas │      │ Object storage│ (blobs, media,
                          └──────────┘      │  (S3/GCS/Blob)│  backups)
                                            └───────────────┘

4.2 The scaling story (a common opening interview prompt)

Senior nuance: don't skip stages. A common, expensive mistake is starting at stage 6 (microservices plus sharding) with 10k users. A single well-tuned Postgres instance handles far more than most startups ever reach; Stack Overflow famously ran on a small number of SQL Server machines for years.

4.3 Load-balancing algorithms

4.4 🏗️ Flow: lifecycle of one request through the stack

 Browser
   │ 1. DNS: resolver → authoritative (GeoDNS / latency-based) → returns anycast VIP
   │ 2. TCP + TLS 1.3 to nearest edge PoP (anycast routes to closest)
   ▼
 Edge / CDN PoP ──cache hit?──yes──▶ respond (≈10-30 ms)
   │ no (dynamic API)
   │ 3. Edge → origin over warm, pooled connections (often a private backbone)
   ▼
 L4 load balancer (ECMP + consistent hashing on 5-tuple, e.g. Maglev-style)
   │ 4. picks an L7 proxy; connection-level, no TLS termination (or TLS passthrough)
   ▼
 L7 proxy / API gateway (Envoy/NGINX): TLS termination, authN, rate limit, routing
   │ 5. route /v1/orders → orders service; picks instance via least-request / P2C
   ▼
 Service instance: deadline = 800 ms remaining
   │ 6. cache GET (0.5 ms) → miss → DB query (3 ms) → cache SET
   │ 7. async: emit event to Kafka (does not block response)
   ▼
 Response flows back; access log + trace span emitted at every hop

4.5 Deeper: load balancers, health checks & connection management

• L4 load balancing at scale: Google's Maglev (NSDI 2016) runs software L4 balancers on commodity machines behind ECMP routers. It uses consistent hashing so that any Maglev machine sends a given connection to the same backend, and adding or removing a balancer doesn't break existing connections. Cloud network LBs are built on the same idea.
• Direct Server Return (DSR): responses bypass the LB and go straight to the client. This helps when responses are much larger than requests (video, downloads).
• L7 at scale: Lyft built Envoy (open-sourced 2016) as a sidecar and edge proxy. It became the data plane for most service meshes. It brought outlier detection (eject hosts with consecutive 5xx), retry budgets, and zone-aware routing into one place.
• Health checks have two jobs: liveness (restart me) and readiness (send me traffic). Never make readiness depend on a shared dependency such as the DB. If the DB blips, every instance reports unready at once and the LB has nowhere to send traffic. Good LBs fail open: when all hosts are unhealthy, route to all of them.
• Slow start: new instances (cold JIT, empty caches) should ramp their share of traffic over 30–120 s. Least-request balancing otherwise floods the "idle" new instance.

4.6 🌍 In production

• Stack Overflow served a top-50 website from a small number of on-prem SQL Server and web servers for years, publicly documented on Nick Craver's blog. Vertical scaling plus heavy caching goes a long way.
• Instagram reached tens of millions of users on Django + Postgres + Memcached + Redis with a very small team, sharding Postgres by hand.
• Shopify kept a modular monolith (Rails), enforcing component boundaries with tooling (Packwerk) rather than splitting into hundreds of services.
• Google Maglev and Lyft Envoy are the canonical L4 and L7 designs to name in an interview.

4.7 ⚠️ Corner cases & deep details