57. Walkthrough: Design a Service Mesh

Step 0 — Clarify the business goal (≈ 3 min)

🎤 Candidate: What's driving this? I can think of three different reasons: (1) security/compliance, zero-trust mTLS everywhere for an audit; (2) reliability, uniform timeouts, retries, and safe canary rollouts; or (3) developer productivity, taking networking boilerplate out of hundreds of codebases in several languages.

🧑‍💼 Interviewer: All three, but the trigger is a compliance requirement: all internal traffic must be encrypted and authenticated within 12 months.

🎤 Candidate: Then mTLS with workload identity is the must-have, and traffic management is the follow-on value. That also tells me the rollout plan matters as much as the architecture, because a mesh that breaks production during adoption gets abandoned. What's the environment?

🧑‍💼 Interviewer: About 3,000 services, 200k pods on Kubernetes across 4 regions, plus some legacy VMs. Polyglot: Java, Go, Python, Node.

🎤 Candidate: Polyglot rules out a library-only approach as the main answer, which points to a proxy-based data plane.

Step 1 — Functional requirements

Step 2 — Non-functional requirements

Step 3 — Estimates

Pods: 200k; services: 3k; endpoints per service: avg ~70 (some 5,000+)
Sidecar memory: ~50-150 MB depending on config size → 200k × 100 MB = 20 TB RAM (!)
Sidecar CPU: ~0.1-0.5 vCPU at moderate RPS → tens of thousands of vCPUs fleet-wide
  → cost of the mesh itself is a top-10 infra line item → config scoping & sidecar-less options matter

Config size if every proxy knows every endpoint: 3k services × 70 endpoints ≈ 210k endpoints
  → each proxy holds all → ~tens of MB per proxy, and every endpoint change fans out to 200k proxies
  Pod churn: deploys + autoscaling ≈ 50 endpoint changes/s fleet-wide
  Naive push: 50 changes/s × 200k proxies = 10M updates/s ← impossible
  With scoping (a proxy only receives services it calls, avg ~20) + batching/debounce (100 ms-1 s)
  + incremental (delta) updates → manageable

Certificates: 200k workloads, 24 h certs renewed at ~50% lifetime → ~4.6 issuances/s average
  BUT: a region-wide restart of 50k pods in 10 min → ~85/s burst → issuance must handle bursts (§56)

🎤 Candidate: The estimates tell me the hard problems are config distribution fan-out and the resource cost of sidecars, not the request path itself.

Step 4 — APIs & data model

The user-facing API is declarative config (Kubernetes CRDs or a config API). The control plane's job is to compile intent into per-proxy config.

# Intent: canary 5% of traffic to v2 with a 2s timeout and safe retries
kind: TrafficPolicy
service: payments
routes:
  - match: {path_prefix: /v1/charge}
    destinations: [{subset: v1, weight: 95}, {subset: v2, weight: 5}]
    timeout: 2s
    retries: {attempts: 2, on: [connect-failure, unavailable], budget_percent: 10}
---
kind: AuthorizationPolicy
service: payments
allow:
  - from: spiffe://corp/ns/checkout/sa/checkout-api
    methods: [POST]
    paths: [/v1/charge]

Internal model (control plane): Service, Endpoint(ip, port, labels, locality, health), Policy(version), ProxyStatus(proxy_id, config_version_acked). The proxy protocol is xDS (Envoy's discovery APIs: LDS listeners, RDS routes, CDS clusters, EDS endpoints, SDS secrets), over long-lived gRPC streams with ACK/NACK per version.

Step 5 — High-level architecture

                         ┌──────────────────── CONTROL PLANE (per region, replicated) ────────────────────┐
  Platform/API ─────────▶│ Config API / CRDs ─▶ Validator (schema, policy lint, blast-radius checks)      │
  (GitOps)               │        │                                                                       │
  K8s API / VM registry ▶│ Discovery (watch endpoints)                                                   │
                         │        ▼                                                                       │
                         │ Config compiler: intent + topology → per-proxy snapshot (scoped)               │
                         │        ▼                                                                       │
                         │ xDS servers (horizontally scaled, sharded by proxy) ── debounce, delta push    │
                         │ CA / identity issuer (SPIFFE IDs) ◀── see §56                                  │
                         └───────────────┬────────────────────────────────────────────────────────────────┘
                                         │ gRPC streams (xDS + SDS)
          ┌──────────────────────────────┼──────────────────────────────────────┐
          ▼                              ▼                                      ▼
   ┌─────────────┐                ┌─────────────┐                        ┌─────────────┐
   │ Pod: app A  │  mTLS (HTTP/2) │ Pod: app B  │                        │ VM: legacy C│
   │  + sidecar  │═══════════════▶│  + sidecar  │                        │ + proxy     │
   │ (Envoy)     │                │ (Envoy)     │                        │ agent       │
   └─────────────┘                └─────────────┘                        └─────────────┘
   iptables/eBPF redirects app traffic into the local proxy (no app code change)
   Telemetry: proxies emit metrics + spans → collectors (§16)

Walk a request: app A calls http://payments/v1/charge → traffic is redirected to A's sidecar → the route table (RDS) matches the path → 95/5 weighted pick of subset v1/v2 → the endpoint (EDS) is chosen locality-first by least-request → mTLS handshake with B's sidecar (certificates from SDS; verify B's SPIFFE ID) → B's sidecar checks the AuthorizationPolicy (is A's identity allowed for POST /v1/charge?) → forwards to app B on localhost → both proxies record metrics and propagate trace headers.

Step 6 — Deep dives

Step 7 — Failure modes, scale, cost, evolution

Cost: the mesh's own footprint (proxy CPU and memory) plus telemetry volume. Track "mesh cost per 1k RPS" as a platform KPI.

Evolution and org: a platform team owns the control plane and proxy versions. Service teams own their TrafficPolicy. Roll out in phases: (1) telemetry only, (2) permissive mTLS, (3) strict mTLS for compliance scope, (4) traffic management for opt-in teams. Adoption metric: % of traffic on mTLS.

Step 8 — Wrap-up

🎤 Candidate: The design is a regional, horizontally scaled control plane compiling declarative intent into scoped, delta-pushed xDS config, and an L4 node-level mTLS data plane with L7 proxies where they add value. Identity is SPIFFE certificates with 24-hour lifetimes. The main risks are the mesh becoming a global failure domain, which we handle with staged config and proxy rollouts and static stability, and its resource cost, which we handle with sidecar-less L4 and scoping. The rollout is permissive → strict per namespace to hit the compliance deadline safely.