55. Walkthrough: Design a Secrets Manager & Key Management Service (KMS)

Step 0 — Clarify the business goal

🎤 Candidate: The outcomes are: no plaintext secrets in code, configs, or images (breach prevention); encryption at rest with auditable key usage (compliance: SOC 2, PCI, customer-managed keys for enterprise deals); and rotation without outages. Two products share the name: a KMS that holds keys and performs crypto operations (keys never leave), and a secrets manager that stores and rotates secrets (DB passwords, API tokens). Both?

🧑‍💼 Interviewer: Both. Internal platform, multi-region, 5,000 services.

Step 1–2 — Requirements

Step 3 — Estimates

5,000 services × ~20 instances = 100k workloads
Envelope encryption: KMS is called to unwrap DATA KEYS, not to encrypt each record.
  With data-key caching (e.g. 5 min TTL, bounded use counts): ~100k workloads × few keys / 300 s
  ≈ ~1-3k KMS decrypts/s steady; cold-start storms (mass restart) → 10-50k/s bursts
HSM throughput is limited and expensive → HSMs protect ROOT keys; most operations use
  root-wrapped intermediate keys in KMS host memory (protected) — the HSM is not on every call
Audit events: ~5k/s → ~0.5B/day → append-only log store

Step 5 — High-level architecture

 ┌───────────────────────── KMS (per region) ─────────────────────────────────────┐
 │ API front end (authN via workload identity mTLS §56, authZ via key policy + IAM)  │
 │   ─▶ Key service hosts (hardened, no SSH, memory-only key material)              │
 │        hold customer master keys (CMK versions) WRAPPED by HSM-resident root keys │
 │        perform Encrypt/Decrypt/GenerateDataKey in memory                          │
 │   ─▶ HSM fleet (FIPS-validated): root/domain keys; unwrap CMKs at load            │
 │   ─▶ Key metadata store (key id, versions, policy, state, region replicas)        │
 │   ─▶ Audit log (append-only, tamper-evident) ─▶ SIEM                               │
 └────────────────────────────────────────────────────────────────────────────────┘
 Envelope encryption by a service:
   1. GenerateDataKey(CMK) → {plaintext DEK, DEK encrypted under CMK}
   2. encrypt data locally with DEK (AES-GCM); store {ciphertext, encrypted DEK, key id/version}
   3. discard plaintext DEK (or cache briefly); to read: Decrypt(encrypted DEK) → DEK → decrypt data
 Secrets manager: secrets stored encrypted (envelope, per-secret DEK) in a replicated DB;
   access via workload identity + policy; versions (AWSCURRENT/AWSPENDING-style staging labels);
   rotation workers; dynamic-secret engines (DB, cloud IAM) issue leased credentials
 Client side: agent/sidecar fetches & caches secrets, renews leases, writes to tmpfs or serves via local API

Step 6 — Deep dives

1. Envelope encryption is the core idea. KMS never sees bulk data. It only wraps small data keys, which gives scale (local encryption), low latency, and a small, auditable operation surface. Key rotation creates a new CMK version: new encryptions use it, old ciphertexts still decrypt with their recorded version, and re-encryption of old data is optional (re-wrap DEKs, not data).
2. No secret zero: workloads authenticate with platform identity (§56 SPIFFE certs, cloud instance identity, Kubernetes service account tokens), never with a bootstrap password baked into images.
3. Zero-downtime secret rotation (e.g. DB password):

   1. create new credential (DB user B or new password) — staged as PENDING
   2. test it works
   3. promote PENDING → CURRENT; clients pick it up on next fetch/lease renewal
   4. keep old credential valid for ≥ max client cache TTL (overlap window)
   5. revoke old credential
   (dual-user pattern: alternate between two DB users so one is always valid)
   

   Dynamic secrets (as in Vault): each workload gets its own short-lived DB credential per lease, so revocation is per workload and leaks are contained and attributable.
4. Availability of a universal dependency: KMS outage means nothing can decrypt, which means everything is down. Mitigations: regional independence (no cross-region calls on the hot path), data-key caching in clients (bounded time and use counts), multi-region replicated keys for DR, static stability (secrets cached in agents survive a secrets-manager outage), per-caller quotas so one service's retry storm can't starve others.
5. Customer-managed keys (BYOK/HYOK): an enterprise tenant's data keys are wrapped by a key in their KMS or HSM. If they revoke it, their data becomes unreadable. That's a sales feature (§17.3, Slack EKM), and an availability dependency on the customer's key service, which must be documented and designed for.

Step 7 — Failure modes & corner cases