40. Load balancing: L4 vs L7, DSR, and connection handling

40.1 L4 (transport) load balancing

Operates on the 5-tuple. Picks a backend per connection and forwards packets; never parses the payload. Fast, protocol-agnostic (works for any TCP/UDP service), preserves end-to-end TLS.

Forwarding modes:

Mode	How	Return path	Notes
NAT / proxy	LB rewrites dst IP (+ maybe src) to the backend	back through the LB	LB sees both directions; backend sees LB's IP as client unless PROXY protocol / `X-Forwarded-For`
DSR (Direct Server Return) / L3-DSR	LB rewrites only dst MAC (same subnet) or tunnels (IPIP/GRE); backend has the VIP on loopback and replies directly to the client	bypasses the LB	LB only handles inbound (often 10% of bytes); backend must not ARP for the VIP; huge scale for video/download
Maglev / ECMP + consistent hashing	routers ECMP-spread flows to a fleet of stateless LB nodes; each hashes the 5-tuple consistently to a backend	via LB or DSR	Google Maglev, Cilium, Katran (XDP). Survives LB node churn without breaking flows.

Connection consistency is the hard part: if the LB set changes (scale, deploy, failure), a flow's packets must keep hitting the same backend. Solutions: consistent hashing (minimal disruption), flow tables synced across LB nodes, or "stickiness" via source-IP hashing.

40.2 L7 (application) load balancing / reverse proxy

Terminates the client TCP+TLS connection, parses HTTP, and opens its own connection(s) to backends. Two independent TCP connections, two congestion-control loops.

Capabilities L4 can't do:

Route by URL path, Host, header, cookie, method.
Connection multiplexing / pooling: thousands of client connections → a small pool of backend keep-alive connections (protects backends from the ephemeral-port / handshake storm — Ch. 21 war story).
Retries, timeouts, circuit breaking, outlier ejection per request.
Protocol translation: HTTP/3 client ↔ HTTP/1.1 backend; gRPC-web ↔ gRPC.
Buffering slow clients (absorb a slow-loris client so the backend isn't tied up — Ch. 52).
Observability: per-request logs, traces, status codes.
Inject X-Forwarded-For / Forwarded / PROXY protocol so backends learn the real client IP.

Cost: it's in the data path for every byte, needs CPU for TLS, and adds a hop of latency + its own buffering/HOL characteristics. Examples: nginx, HAProxy, Envoy, Traefik, ALB, Cloudflare.

40.3 Health checks

L4 check: can I connect() to backend:port? (TCP SYN/ACK). Cheap, but "port open" ≠ "app healthy."
L7 check: GET /healthz → expect 200 and maybe a body. Reflects real readiness (DB reachable, caches warm).
Passive / outlier detection: eject a backend that returns 5xx / connection errors for real traffic, probe it, re-admit.
Failure math: check interval × unhealthy-threshold = detection time. Too aggressive → flapping under load; too slow → users hit dead backends. Pair with fast client retries to a different backend.

40.4 Session persistence ("sticky sessions")

Route a given user to the same backend (for in-memory session state):

Source-IP hash (L4) — breaks with NAT/CGNAT (many users → one backend) and mobile IP changes.
Cookie-based (L7) — LB sets/reads a cookie naming the backend. Robust.
Better: externalize session state (Redis, signed JWT) so any backend can serve any request and you don't need stickiness at all.

40.5 Draining and deploys

To remove a backend without dropping requests: mark it draining → LB stops sending new connections/requests, existing ones finish (bounded by a drain timeout), backend sends HTTP/2 GOAWAY / closes keep-alives gracefully, then it's pulled. Skipping drain = a burst of 502/connection reset on every deploy.

40.6 War story

A service behind an L7 proxy saw periodic 502 Bad Gateway spikes exactly at deploy time. The proxy kept a pool of keep-alive connections to backend pods; when Kubernetes killed a pod, the proxy sent a request onto an already-half-closed connection and got a RST → 502. Fixes: (1) backend sends Connection: close / HTTP/2 GOAWAY and a preStop sleep so the proxy observes readiness=false and drains first; (2) proxy configured to retry idempotent requests once on a connection error to another backend. 502s went to zero. Lesson: L7 LBs pool backend connections; backends must announce shutdown (GOAWAY/close) and LBs must retry connection-level errors, or every deploy sheds requests.

Try it yourself

# Which backend am I hitting? (if the app echoes hostname)
for i in $(seq 10); do curl -s https://example.com/whoami; done
# See the LB's added headers:
curl -s https://example.com/headers | grep -iE 'x-forwarded|forwarded|via|x-real-ip'
# L4 vs L7 clue: does end-to-end TLS work (L4) or does the cert belong to the LB (L7)?
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>&1 | grep -E 'issuer|subject'
# HAProxy/Envoy admin stats if exposed:
curl -s http://lb:8404/stats\;csv   # HAProxy
curl -s http://lb:9901/clusters      # Envoy