56. Walkthrough: Design a Certificate Authority / Cert-Minting Service

The prompt. "Design a service that issues TLS certificates to every service in our company." Variants: "Design an internal PKI", "Design Let's Encrypt", "Design workload identity for a zero-trust network", "Design a secrets/certificate management platform". It's a security infrastructure problem that tests trust bootstrapping, key protection, availability under expiry deadlines, revocation, and safe rotation. Expect it at Google, Microsoft (Azure/Entra), cloud providers, and any platform/security team. The service mesh in §57 depends on it.

Steady state: 1M certs × renew every 12 h (50% of 24 h) → 1M / 43,200 s ≈ 23 signs/s
Bursts: mass restart of 100k pods in 10 min → ~170/s; full-fleet emergency reissue in 4 h → ~70/s
        → design for ~500-1,000 signs/s per region (headroom)
Signing cost: ECDSA P-256 sign ≈ tens of µs/core in software; RSA-2048 ≈ ~1 ms/core
  HSM / cloud KMS: throughput is limited and model-dependent (often hundreds to a few thousand ops/s
  per device or per key quota) and adds ms of latency → multiple intermediates/HSMs per region, or
  KMS-backed intermediates with quota increases; ECDSA preferred
Records: 2M issuances/day × ~2 KB (cert + metadata) ≈ 4 GB/day ≈ 1.5 TB/year → modest
Audit log: same volume, append-only, Merkle-tree hashed
Trust bundle: a few KB, but must reach 1M workloads within minutes during rotation

Workload API (local, on every node — Unix domain socket, no network credentials needed):
  FetchX509SVID()   stream → {spiffe_id, cert_chain, private_key(local), bundle}   (pushed on rotation)
  FetchTrustBundle() stream → {roots[], version}

Node agent → Issuer (mTLS with node identity):
  POST /v1/svid:sign {csr, workload_selectors, node_attestation_ref} → {cert_chain, ttl}

Admin / policy (GitOps, reviewed):
  POST /v1/registrations {selectors:{k8s:ns=payments, k8s:sa=payments-api, cluster=prod-eu1},
                          spiffe_id:"spiffe://corp/prod/payments-api", dns_sans:["payments.internal"],
                          max_ttl:"24h"}
  POST /v1/revocations {serial | spiffe_id | node_id, reason}
  GET  /v1/inventory?expiring_within=72h

Public (phase 2): ACME (RFC 8555): newAccount → newOrder → challenges (HTTP-01 / DNS-01 / TLS-ALPN-01)
  → finalize(CSR) → certificate

 ┌─────────────────────────── OFFLINE (air-gapped vault) ───────────────────────────┐
 │ Root CA key in HSM · used ~yearly in a recorded M-of-N key ceremony              │
 │ signs: regional intermediates (name-constrained: spiffe://corp, *.internal)       │
 └──────────────────────────────────────────────────────────────────────────────────┘
                     │ intermediate certs (keys generated inside regional HSM/KMS)
 ┌───────────── REGION (×4, independent) ───────────────────────────────────────────┐
 │  Issuer service (stateless, N replicas, LB)                                       │
 │    1. authenticate node agent (mTLS with node SVID)                               │
 │    2. check node is attested & not banned; check workload selectors are           │
 │       scheduled ON THAT NODE (node can only request identities it hosts)          │
 │    3. match registration entry → identity, SANs, max TTL (policy engine)          │
 │    4. build TBS certificate (serial, notBefore = now − 5 min, notAfter ≤ policy)  │
 │    5. sign via HSM/KMS (intermediate key)                                          │
 │    6. append to audit log (must succeed before returning — "log then issue")       │
 │  Policy DB · Issued-cert store · Audit log · Revocation/deny-list publisher       │
 └───────────────────────────────▲──────────────────────────────────────────────────┘
                                 │ mTLS
 ┌──────────── NODE (every host / k8s node) ─────────┐
 │ Node agent:                                        │
 │   boot: NODE ATTESTATION (cloud identity doc/TPM)  │──▶ Issuer: verifies with cloud provider / TPM CA
 │   per workload: WORKLOAD ATTESTATION               │
 │     (pid → cgroup → pod UID → kubelet → ns/sa)     │
 │   generates workload key pair locally, sends CSR   │
 │   caches certs; renews at 50% TTL with jitter      │
 │   serves Workload API over Unix socket ───────────▶│ app / mesh proxy (SDS) gets cert + key + bundle
 └────────────────────────────────────────────────────┘
 Expiry inventory scanner (all certs, incl. legacy ones not issued by us) → alerts & dashboards
 Transparency monitors → alert on unexpected identities (e.g. a new cert for payments from an odd node)

Deep dive 1: trust bootstrapping (the "secret zero" problem)

🧑‍💼 Interviewer: How does a brand-new VM prove who it is before it has any credentials?

🎤 Candidate: With evidence signed by something we already trust, never a long-lived secret baked into an image:

• Cloud VMs: the instance identity document or token signed by the cloud provider (AWS instance identity document, GCP instance identity JWT, Azure IMDS attested data). The Issuer verifies the signature and checks the instance ID is real, running, in our account, and hasn't attested before (prevents replay).
• Bare metal: a TPM quote (hardware-rooted key plus measured boot).
• Kubernetes: projected service-account tokens (audience-bound, short-lived) validated against the cluster's issuer.

That gives the node an identity. The node agent then attests workloads locally from kernel facts. The crucial authorization rule is that a node can only obtain identities for workloads scheduled on it. A compromised node can impersonate only what it already runs, not the whole company.

Deep dive 2: key hierarchy and protection

 Root (offline, HSM, 10-20 y)  ──signs──▶  Intermediate per region (online, HSM/KMS, 1-3 y,
                                            name-constrained, pathLen=0)
                                                 ──signs──▶ Workload certs (24 h, no CA bit)

• Root offline: compromise of an online system can't touch the root. Key ceremonies need M-of-N custodians (e.g. 3 of 5 smartcards), are recorded, and are scripted.
• Intermediates per region: a regional compromise or outage stays regional, and each region's HSM capacity scales independently.
• Name constraints on intermediates: even if one is misused, it can only sign names in our internal namespaces, never bank.com.
• Algorithm agility: ECDSA P-256 today (fast, small chains). Design the profile system so the algorithm is configuration, to allow a move to post-quantum or hybrid signatures without redesign.

Deep dive 3: making expiry impossible to turn into an outage

 Lifetime L = 24 h; renew at L/2 = 12 h (± jitter 10%); retry with backoff on failure
   → CA outage tolerance ≈ 12 h before the first certs expire
 Alert levels: renewal failing for 1 h → ticket; cert age > 75% of L → page; any cert < 2 h → SEV
 Shorter L = smaller compromise window, less need for revocation, LESS outage tolerance
 Longer L  = more outage tolerance, larger compromise window, MORE reliance on revocation

• Static stability: agents and proxies keep using the current cert if the Issuer is unreachable, and agents cache bundles. No component on the request path calls the CA.
• Jitter notAfter and renewal times so the fleet doesn't renew in synchronized waves (for example after a mass restart, when everyone's certs would otherwise expire at the same minute).
• Multi-region issuers: an agent can fail over to another region's issuer, which signs with that region's intermediate. All intermediates chain to the same root, so validation still works.
• Inventory of all certificates, including ones we didn't issue (legacy appliances, vendor-managed, public): scan endpoints, parse expiry, and assign an owner. Most expiry outages come from certs outside the automated system.

Deep dive 4: revocation

Intermediate compromise is the nightmare scenario: revoke the intermediate, remove it from trust bundles, and re-issue every certificate it signed. With 24 h certs and automated renewal, that's "trigger renew-now for the fleet" at the capacity sized in Step 3 (≈ 4 h). Rehearse it as a game day.

Deep dive 5: CA rotation without breaking the fleet

 1. Generate new intermediate (or new root in a ceremony)
 2. Distribute trust bundle {old, new} to ALL relying parties → measure convergence (≥ 99.99%)
 3. Start issuing from the new intermediate (canary region first)
 4. Wait ≥ max cert lifetime (+ margin) → all live certs now chain to new
 5. Remove old from bundles → monitor handshake failures by "unknown CA"
 Never invert steps 2 and 3: issuing from a CA that clients don't yet trust = mass outage.

Deep dive 6 (phase 2): what changes for public certificates

• Domain validation via ACME challenges: HTTP-01 (a file at /.well-known/acme-challenge/), DNS-01 (a TXT record, needed for wildcards), TLS-ALPN-01. Multi-perspective validation: validate from several network vantage points so a local BGP hijack can't fool the CA.
• CAA records: the domain owner restricts which CAs may issue. Check them at issuance time.
• Certificate Transparency: submit precertificates to public CT logs and embed the signed timestamps (SCTs). Browsers require it. Monitor CT for certificates issued for your domains by anyone.
• Compliance: the CA/Browser Forum Baseline Requirements, annual audits, serial entropy, strict incident reporting, and revocation deadlines (24 h for key compromise, 5 days for many other issues). These deadlines are why public CAs need mass-revocation capacity.
• Rate limits per account and per registered domain protect the CA from abuse and runaway clients.
• Build vs buy: for our own customer-facing domains we don't become a public CA (audits, compliance staff, root program inclusion take years). We automate issuance from public CAs via ACME, with redundancy across two CAs in case one has an incident.

Interviewer follow-ups & strong answers

🌍 Real-world references

• Let's Encrypt (the ISRG): the largest public CA by volume, issuing millions of certificates per day with automation only (ACME, RFC 8555). Its open-source Boulder CA splits responsibilities into separate components (web front end, registration authority, validation authority, CA signer, storage authority), which is a good model for least privilege between stages. Notable events: the March 2020 CAA-checking bug, which forced mass revocation under Baseline Requirements deadlines, and the September 2021 expiry of the old "DST Root CA X3", which broke older devices that didn't trust the newer root. That's Deep dive 5 in real life.
• SPIFFE / SPIRE (CNCF): the open standard and reference implementation for the node-attestation + workload-attestation + Workload API design shown here.
• Google ALTS (Application Layer Transport Security whitepaper): Google's internal mutual authentication and encryption for RPCs, based on service identities rather than hostnames.
• Netflix Lemur (open-sourced 2015): a TLS certificate management and orchestration tool. Cloudflare CFSSL and the 2015 post "How to build your own public key infrastructure" describe Cloudflare's internal PKI.
• Managed options: AWS Private CA, Google Certificate Authority Service, Azure Key Vault certificates, HashiCorp Vault PKI. Often the right "buy" answer for the CA core, with your own attestation and policy layer on top.
• Expiry outages (why this exists): Microsoft Teams (February 2020, an expired authentication certificate); Ericsson's expired certificate in network software (December 2018), which disrupted O2 UK and SoftBank mobile data; and Equifax (2017), where an expired certificate on a traffic-inspection device meant the data exfiltration wasn't seen for months. Expired certificates break security monitoring, not just availability.
• CA/Browser Forum ballot (2025): a phased reduction of public TLS certificate maximum validity down to 47 days by 2029. That's the business driver for automation everywhere.

⚠️ Corner cases

• Low entropy at early boot: VMs generating keys before the RNG is seeded. Use the OS CSPRNG (getrandom blocks until seeded) and never generate keys in image-build pipelines.
• Private keys leaking through core dumps, logs, or debug endpoints: keep keys in memory only, with redaction and locked memory where possible.
• Chain size and QUIC: large chains (RSA plus several intermediates) can exceed QUIC's anti-amplification limit (the server may send only 3× the bytes received before address validation), forcing extra round trips. Use ECDSA and short chains.
• Serial number entropy: public CAs must use ≥ 64 bits from a CSPRNG. Collisions or predictability are compliance incidents.
• Hostname validation by CN: modern clients match only SANs. Certs with the name only in CN fail in modern TLS stacks.
• Synchronized expiry: a mass restart puts the whole fleet on the same schedule. Jitter both notAfter and renewal times.
• Time-of-check vs time-of-issue: a policy revoked between the policy check and signing still issues one cert. Bind issuance to a policy version, and monitor through the transparency log.
• Clients that cache trust bundles forever: old mobile apps and devices pin roots. Plan multi-year overlaps, or a separate hierarchy for devices you can't update.