54. Walkthrough: Design a Global Configuration & Feature-Flag Platform

Step 0 — Clarify the business goal

🎤 Candidate: What outcomes are we after: release velocity (decouple deploy from release), experimentation (A/B tests drive product decisions), or incident mitigation (kill switches)? All three shape the design.

🧑‍💼 Interviewer: All three. Also, last quarter two of our largest outages were config changes.

🎤 Candidate: Then the platform must make changes fast (kill switch in seconds) and safe (staged, validated, reversible) at the same time. The safety features are the core of the product, not add-ons.

Step 1–2 — Requirements

Step 3 — Estimates

Server instances: 1M; flag snapshot per service ≈ 50 KB-5 MB
Changes: ~2k/day (bursty around release hours)
Naive push of full snapshots: 1M instances × 1 MB × 2k changes/day = 2 PB/day ← no
  → per-service scoping + deltas + fan-out tree with local caches
Evaluations: ~10B/s fleet-wide → must be local, in-memory, lock-free reads
Mobile/web: 500M devices; can't hold 500M streaming connections cheaply
  → CDN-cached, environment-level snapshots fetched on app start/foreground + periodic poll;
    streaming only for active sessions that need fast kill switches

Step 5 — High-level architecture

 Admin UI / API / Git (config-as-code) ─▶ Validation: schema, lint, DRY-RUN evaluation
     ("this change flips 100% of prod traffic" → requires approval), conflict checks
   ─▶ Approval workflow (2-person for prod critical flags) ─▶ Config store (strongly consistent,
     versioned, append-only history; Git or Spanner/etcd-like)
   ─▶ Publisher ─▶ regional relays (fan-out tree) ─▶ per-host agent (on-disk cache, last-known-good)
                                                     ─▶ SDK in-process (memory, atomic swap of snapshot)
   ─▶ Edge snapshot service (CDN) ─▶ mobile/web SDKs (poll / SSE for active sessions)
 SDK exposure events ─▶ Kafka ─▶ experimentation analytics
 Progressive delivery controller: apply change to cell 1 / 1% → watch SLOs → expand or auto-revert

Step 6 — Deep dives

1. Deterministic bucketing: bucket = hash(flag_key + salt + user_id) % 100000. The same user always gets the same variant on every server, and increasing a rollout from 5% to 20% keeps the original 5% in (monotonic). Changing the salt reshuffles everyone, which is an intentional act only for new experiments.

2. Static stability: the SDK loads the last-known-good snapshot from the local agent or disk at startup and never blocks startup on the config service. Compiled-in defaults are the final fallback. If the platform is down, nothing changes, which is the right failure mode.

3. Change safety (the main point): treat config like code:

   • Review and CI validation.
   • Canary by cell or percentage with automated health checks against SLOs.
   • Automatic rollback.
   • A change freeze during incidents (except kill switches).
   • Every change linked to an owner and a ticket.

   This is what turns "config caused our two biggest outages" into a solved problem.

4. Consistency across services: one user request crosses 5 services that receive a flag update at slightly different times, so the user sees a mixed experience for a few seconds. Usually acceptable. For experiments, pass the evaluated variant in the request context so downstream services use the same assignment.

5. Flag debt: every flag has an owner and an expiry date. CI warns on stale flags, and dead-code detection removes fully rolled-out flags (Knight Capital, §29.6).