Chapter 9 — Load Balancers & Rate Limiting

Netflix has thousands of servers. A load balancer decides which server handles your request — and makes sure no single server gets overwhelmed. Rate limiting makes sure no single user overwhelms Netflix.

Why Load Balancers Exist

One server can handle perhaps 1,000 concurrent connections. Netflix has 200+ million subscribers. Solution: many servers, each handling a slice of traffic.

Without load balancer:
  All users → netflix.com:443 → 1 server → overloaded → down ✗

With load balancer:
  All users → netflix.com:443 → Load Balancer
                                  ├──► Server 1  (handles 1,000 users)
                                  ├──► Server 2  (handles 1,000 users)
                                  ├──► Server 3  (handles 1,000 users)
                                  └──► Server N  (handles 1,000 users)
  N servers → N × 1,000 concurrent users ✓

L4 Load Balancer — Transport Layer

An L4 load balancer operates at the transport layer (TCP/UDP). In a pass-through design, it sees IP addresses and ports but does not decrypt TLS or read HTTP headers.

Client → L4 LB:443 → Backend:443 (TCP stream forwarded as-is)
             ↑
  L4 LB reads: SrcIP, DstIP, SrcPort, DstPort
  L4 LB does NOT read: HTTP method, URL, headers, cookies

How L4 Load Balancing Works

Method 1: NAT (D-NAT) The load balancer rewrites the destination IP:

Packet: Dst=52.38.36.82:443  →  LB rewrites  →  Dst=10.0.4.55:443
Return: Src=10.0.4.55:443   →  LB rewrites  →  Src=52.38.36.82:443
LB is in the path of every packet.

Method 2: Direct Server Return (DSR) The load balancer only handles the inbound SYN, then steps aside:

Client → LB (initial SYN)
LB → Backend (forwards connection, sets up rules)
Backend → Client DIRECTLY (response bypasses LB)
LB is only in the path of inbound packets. Outbound bypasses it.
Much faster for high-bandwidth responses (video!).

L4 Load Balancing Algorithms

Round Robin:

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A  (cycles)
Simple. Ignores server load.

Weighted Round Robin:

Server A weight=5 → gets 5 requests per cycle
Server B weight=3 → gets 3 requests per cycle
Server C weight=2 → gets 2 requests per cycle
Use when servers have different capacity.

Least Connections:

Server A: 100 active connections
Server B: 50 active connections  ← WINNER (fewest)
Server C: 75 active connections
New request → Server B
Best for requests with variable processing time.

IP Hash:

Hash the client's source IP → maps to a backend while the pool is unchanged
hash(203.0.113.45) % 3 = 1 → Server B
Same client tends to hit the same server until the backend set or hash policy changes.
Breaks if a server is added or removed (consistent hashing fixes this).

Consistent Hashing:

Servers are placed on a "virtual ring" (hash ring).
Each request's key (IP, user ID) is hashed to a point on the ring.
Request goes to the nearest server clockwise on the ring.

Adding/removing a server only remaps a fraction of requests
(vs IP hash which remaps all requests).
Used by Netflix's Cassandra, Redis, and CDN caches.

L4 LB Examples

• AWS NLB (Network Load Balancer)
• HAProxy (in TCP mode)
• Linux LVS (IP Virtual Server) — used by Netflix
• IPVS (IP Virtual Server — kernel-level, extremely fast)

L7 Load Balancer — Application Layer

An L7 load balancer operates at the application layer. It decrypts TLS, reads HTTP headers, URLs, cookies — and makes routing decisions based on content.

Client ──[HTTPS]──► L7 LB (TLS terminated here)
                       │  reads HTTP: method, path, headers, cookies
                       ├──► API servers  (if path starts with /api/)
                       ├──► Static servers  (if path ends with .js/.css)
                       ├──► Stream servers  (if path starts with /video/)
                       └──► Auth servers  (if path is /login)

What L7 LBs Can Do That L4 Cannot

L7 Content-Based Routing

Incoming request: GET /api/v1/users/me
  → Rule: path starts with /api/ → route to API cluster

Incoming request: GET /static/app.abc123.js
  → Rule: path ends with .js → route to CDN / static servers

Incoming request: GET /browse
  → Rule: path is /browse → route to UI rendering cluster

Incoming request: GET /video/stream/12345
  → Rule: path starts with /video/ → route to streaming cluster

Sticky Sessions (Session Affinity)

Some applications store session state on the server. Those systems may require the same user to keep reaching the same server, or they need the session state moved to shared storage.

Method 1: Cookie-based (L7 only)
  First request → Server A → LB sets cookie: SERVERID=A
  Next requests → LB reads cookie → routes to Server A

Method 2: IP-hash (L4 or L7)
  Same client IP → same server (best-effort)
  Breaks if server fails or client changes IP

Best practice: Use distributed session stores (Redis) instead.
  Every server can serve any user — no stickiness needed.

L7 LB Examples

• Nginx (web server + reverse proxy + L7 LB)
• Envoy Proxy
• AWS ALB (Application Load Balancer)
• HAProxy (in HTTP mode)
• Netflix Zuul (their own JVM-based L7 gateway)

Health Checks — Removing Dead Servers

The load balancer continuously checks if each backend is healthy. Unhealthy backends are removed from rotation until they recover.

Types of Health Checks

TCP Health Check (L4):

LB → Tries to open TCP connection to Backend:8080
Connection SUCCESS → healthy ✓
Connection REFUSED/TIMEOUT → unhealthy ✗
Doesn't verify the app is actually working — just that the port is open.

HTTP Health Check (L7):

LB → GET /health HTTP/1.1 Host: backend
Backend → 200 OK {"status": "healthy", "db": "ok", "cache": "ok"}
LB checks: response code == 200? body contains "healthy"? ✓

If backend returns 500 or times out → marked unhealthy → removed from pool

Deep Health Check:

GET /health/deep
Backend checks all dependencies:
  - Database connection: OK
  - Redis cache: OK
  - Downstream services: OK
Returns 200 only if everything is working.
Prevents routing to a server that is "up" but functionally broken.

Health Check Configuration

Healthy threshold:    2 consecutive successes → mark healthy
Unhealthy threshold:  3 consecutive failures  → mark unhealthy
Check interval:       5 seconds
Check timeout:        2 seconds

Timeline:
  t=0:   Backend healthy, serving traffic
  t=10:  Backend fails (disk full)
  t=15:  Health check 1: FAIL
  t=20:  Health check 2: FAIL
  t=25:  Health check 3: FAIL → marked UNHEALTHY → removed from LB pool
  t=30:  No more traffic sent to this backend ✓

  t=60:  Disk cleared, backend recovers
  t=65:  Health check 1: SUCCESS
  t=70:  Health check 2: SUCCESS → marked HEALTHY → added back to LB pool ✓

Rate Limiting

Rate limiting controls how many requests a client can make in a given time period. Prevents:

• Abuse (credential stuffing, scraping)
• Accidental DDoS from buggy clients
• One user consuming all resources (noisy neighbor)

Rate Limiting Algorithms

Token Bucket:

A bucket holds tokens. Each token = permission to make 1 request.
Bucket refills at a steady rate. Bucket has a max capacity (burst).

Capacity: 100 tokens
Refill rate: 10 tokens/second

At 10 req/s: steady state — bucket stays full, requests allowed
At 50 req/s: burning through 40 tokens/s, bucket empties in 2.5s → throttled
At 1 req/s:  tokens accumulate, can burst later

Allows SHORT BURSTS above the sustained rate.

Leaky Bucket:

Requests go IN to a bucket.
Bucket drains (processes) at a fixed rate.
If bucket FULL → reject new requests.

Rate: 100 req/s exactly.
Smooth, predictable output rate.
No bursting allowed.
Good for protecting backend capacity.

Sliding Window Counter:

Track requests in the last N seconds.
Count requests in [now - N, now].
If count >= limit → reject.

More accurate than fixed windows (which can allow 2x the limit at window boundaries).

Fixed Window Counter:

Count requests in the current window (e.g., this minute).
If count >= limit in this window → reject.
Reset at next window boundary.

Simple but can be gamed: 100 req at 12:00:59, 100 req at 12:01:00 = 200 req/s burst.

Rate Limiting Layers at Netflix

Layer 1: L3/L4 — Packet-level (iptables / eBPF)
  Rule: DROP if source IP sends > 100,000 packets/sec
  Purpose: DDoS mitigation — stop volumetric attacks before they hit the app
  Speed: Line rate (millions of packets/sec evaluated)

Layer 2: L7 — Request-level (Nginx / Envoy)
  Rule: 100 requests/second per IP on /api/login
  Rule: 1,000 requests/minute per authenticated user
  Purpose: Prevent credential stuffing, scraping, API abuse
  Response: HTTP 429 Too Many Requests
  Headers: Retry-After: 30  (try again in 30 seconds)

Layer 3: Application — Business logic
  Rule: Max 5 concurrent streams per account
  Rule: Max 3 failed login attempts → CAPTCHA
  Rule: Max 1 password reset email per hour
  Purpose: Enforce service limits, prevent account abuse

Distributed Rate Limiting

A single server can rate-limit locally. But if you have 100 servers and a client hits a different one each request, each server sees only 1% of the traffic — local rate limiting fails.

Solution: Shared state via Redis:

Each Nginx server:
  On request → INCR user:12345:requests in Redis (atomic)
  Check: if count > 1000 → return 429
  TTL on key: 60 seconds (sliding window)

All servers share the same counter → correct rate limiting regardless of which server handles the request.

Circuit Breaker — Failing Fast

A circuit breaker prevents cascading failures when a backend service is struggling.

CLOSED state (normal):
  Requests flow to backend.
  If error rate > 50% in last 10s → trip to OPEN.

OPEN state (backend is failing):
  Requests immediately return error (503) without hitting backend.
  Backend gets time to recover.
  After 30s → move to HALF-OPEN.

HALF-OPEN state (testing recovery):
  Let 1 request through.
  SUCCESS → return to CLOSED.
  FAILURE → return to OPEN (wait another 30s).

Why this helps:
  Without circuit breaker: All requests pile up waiting for slow backend.
  With circuit breaker: Fail fast → caller can retry elsewhere or return cached data.
  Backend isn't bombarded with traffic while it's struggling → recovers faster.