Frontend Labs
⋯

Chapter 44 — Key management lifecycle

In one sentence

A cryptographic key is only as good as its weakest lifecycle stage — generation, distribution, rotation, escrow, and destruction — and getting any one of them wrong (a predictable key, a leaked key, a key nobody can rotate, a backdoor escrow, a key that survives "deletion") undoes every other control in this Part.

Where we are

Part 2 Ch 9 introduced KMS and envelope encryption; Ch 42 used per-tenant keys for field encryption. This chapter is the operational discipline around every key in the system, independent of which cloud or algorithm.

How it works

The lifecycle:

   GENERATION
     always a CSPRNG (G1 Ch 12), ideally inside an HSM/KMS that never
     exports the raw key material for root/master keys. For BYOK
     (bring-your-own-key), generate offline with a real entropy source
     and import via the provider's secure import ceremony (wrapped with
     the provider's public wrapping key -- never transmitted in the
     clear).

   THE HIERARCHY (recap and extend Part 2 Ch 9):
     ROOT / MASTER KEY (HSM-backed, rarely touched, the ultimate trust
       anchor -- e.g. your KMS's root of trust, or an offline root CA key
       for signing, G1 Ch 29)
       -> INTERMEDIATE / CUSTOMER MASTER KEYS (per environment, per
          tenant -- these are what you actually reference in policy)
          -> DATA ENCRYPTION KEYS (ephemeral, per-object/per-field,
             wrapped by the CMK, never stored unwrapped)
     Splitting the hierarchy this way means ROTATING the top of the tree
     is cheap (re-wrap the layer below; you don't have to touch the
     bulk data) while the ROOT changes rarely and under heavy control.

   DISTRIBUTION
     never transmit a raw key over an unauthenticated or unencrypted
     channel. Envelope encryption (Part 2 Ch 9) is itself a distribution
     mechanism -- the DEK travels ONLY in its wrapped form; only the KMS
     call unwraps it, in memory, for the instant it's needed.

   ROTATION -- two different things people conflate:
     BACKING-KEY ROTATION ("rotate the CMK"): the KMS creates a NEW key
       VERSION under the same key ID; NEW encrypt/wrap operations use the
       new version; OLD ciphertext remains decryptable because the KMS
       keeps old versions available. Cheap, usually automatable
       (annual auto-rotation is a checkbox in most cloud KMS), and does
       NOT require touching your bulk data.
     DATA RE-ENCRYPTION ("actually re-encrypt everything under a new
       key"): required when you suspect the OLD key material itself is
       compromised (not just "it's been a year"), or a compliance
       standard mandates a hard cryptoperiod. Expensive -- you must
       decrypt-then-re-encrypt every object/record. Plan for it (batch
       jobs, versioned ciphertext format with a key-id prefix so you can
       migrate incrementally) rather than discovering you can't do it
       during an incident.
     Because DEKs are generated per-object/per-field and wrapped by the
     CMK, "rotating the CMK" (cheap) already limits the blast radius of a
     CMK compromise going forward -- new data uses the new version
     immediately. It's the OLD data, still wrapped by the old version,
     that needs the expensive path if that old version is the one that
     leaked.

   ESCROW AND BACKUP -- deliberately, not accidentally:
     Disaster recovery needs SOME way to recover keys if a region/HSM is
     lost -- multi-region KMS replication, or an offline backup of a root
     key under strict M-of-N control (below). This is legitimate and
     necessary.
     The RISK version of the same idea is a backdoor: a key escrowed
     "for support purposes", a master key one person can export, a
     vendor with unilateral key access. Every escrow path must be
     justified, access-controlled as tightly as the key itself, and
     logged -- treat "can this key be recovered by someone other than an
     authorized custodian following a documented procedure" as a
     standing audit question.

   DESTRUCTION / CRYPTO-SHREDDING
     see Part 2 Ch 9 and Ch 46 -- destroying the key that wraps a dataset
     makes that dataset permanently unrecoverable, which is how you
     satisfy erasure requests and tenant offboarding across every copy
     (including backups) simultaneously, PROVIDED no plaintext copy or
     unwrapped-key copy exists elsewhere (verify this, don't assume it).

   ALGORITHM AGILITY
     key metadata should record the ALGORITHM and PARAMETERS it was
     created under (G1 Ch 25's crypto-agility principle, applied to keys
     specifically) so a future migration (e.g. a post-quantum signing key
     rotation) is a planned rollover, not an archaeology project.

HSM operations and key ceremonies (for root/master keys):

   FIPS 140-2/3 Level 2 or 3 hardware for root keys -- tamper-evidence
   (L2) or tamper-response (L3), so key extraction requires defeating
   physical protections, not just software access.

   SPLIT KNOWLEDGE / DUAL CONTROL / M-of-N: no single person can
   reconstruct a root key alone. A KEY CEREMONY generates the key inside
   the HSM (it never exists outside it) and splits any BACKUP/recovery
   material into N shares held by different custodians, requiring M of
   them (e.g. 3 of 5) to reconstruct -- and reconstruction itself happens
   inside a controlled ceremony, logged, witnessed, rare.
   (This is exactly how DNSSEC root-zone key ceremonies and most
   certificate authority root-key generations work -- publicly documented
   processes worth reading once for the model.)

   SEPARATION OF DUTIES: whoever administers KEYS (creates, rotates,
   sets policy) should not be the same role that administers the DATA
   those keys protect, and vice versa. A single compromised admin account
   then can't both access the data AND grant itself the key to decrypt
   it.

   MONITORING: every key USE (encrypt/decrypt/sign) and every key ADMIN
   action (create, rotate, policy change, delete) is logged (Part 2 Ch 9's
   CloudTrail example) and reviewed; a spike in decrypt calls for one key,
   or a policy change outside a change window, is a detection rule
   (Ch 54), not just an audit artefact.

   BYOK vs HYOK vs cloud-managed:
     cloud-managed key       simplest; provider generates and holds it.
     BYOK (bring your own)   you generate the root key material (often in
                              your own HSM) and import it into the
                              provider's KMS -- you control generation,
                              the provider operates the service.
     HYOK (hold your own /
       external key manager)  the key material NEVER enters the
                              provider's environment; an EXTERNAL KEY
                              STORE (AWS XKS, GCP External Key Manager)
                              on YOUR infrastructure services
                              encrypt/decrypt calls -- maximum control,
                              maximum operational burden, and a hard
                              dependency: if your external KMS is down,
                              the cloud provider cannot decrypt ANYTHING
                              that depends on it, including their own
                              support access.

Worked example

A CMK-compromise response, showing why the hierarchy matters.

   INCIDENT: an over-permissioned IAM role that had `kms:Decrypt` +
   `kms:GenerateDataKey` on the `prod-orders` CMK is discovered to have
   been usable by a compromised CI job for the last 30 days (Part 3
   Ch 12's OIDC-trust-misconfiguration scenario, realised).

   RESPONSE, using the lifecycle:
     1. IMMEDIATE: revoke the role's access to the key (update the key
        policy); this stops FUTURE encrypt/decrypt calls with that
        identity, but does nothing about ciphertext already exposed if
        the DEKs were exfiltrated (they generally aren't -- envelope
        encryption keeps the DEK's plaintext form transient and in-
        process, Part 2 Ch 9 -- but you must verify no logging/caching
        captured it).
     2. BACKING-KEY ROTATION: rotate `prod-orders` to a new key version
        immediately. Cheap, automatic. All NEW data is now wrapped by a
        version the compromised role never had a session for (a session's
        temporary credentials also expire on their own, Part 2 Ch 5-6).
     3. ASSESS: pull CloudTrail for every `Decrypt`/`GenerateDataKey` call
        by that role in the 30-day window -- the ENCRYPTION CONTEXT
        (Part 2 Ch 9) on each call tells you exactly WHICH tenant's data
        was actually decrypted (not just "the key was used" -- which
        record). This is only possible because you set a meaningful
        encryption context per call.
     4. SCOPE: for the specific tenants/records confirmed decrypted
        during the compromise window, this is now a data breach requiring
        the G1 Ch 51 incident-response and notification process --
        scoped precisely because of step 3, not "notify everyone
        because we can't tell."
     5. DATA RE-ENCRYPTION: NOT required for the whole dataset -- only the
        OLD key version is potentially exposed, and only for records
        confirmed accessed; if a full re-encryption is still warranted by
        policy, you now have the versioned-ciphertext migration path
        (above) ready rather than improvised.
     6. LONG TERM: fix the IAM/OIDC misconfiguration (Part 3 Ch 12),
        require encryption-context-per-call org-wide as a policy, and add
        the "decrypt volume by role" detection rule (Ch 54) that should
        have caught this in days, not 30.

Practice (60 min)

  1. In your sandbox, enable automatic annual key rotation on a customer- managed KMS key; read the docs on exactly what rotation does and doesn't re-encrypt.
  2. Design (on paper) your key hierarchy for SecureShop: root -> per-environment CMKs -> per-tenant CMKs -> per-record DEKs. Write down who/what can administer each layer and who/what can only use each layer (separation of duties).
  3. Simulate the incident above: revoke a role's key-policy access, rotate the key version, and query your audit log filtered by encryption context to scope exactly what was exposed.
  4. Research (don't necessarily implement) BYOK for one of your KMS keys — what would change operationally, and what new failure mode (losing your own HSM) would you now own?
  5. Write a one-page "key ceremony" procedure for your organisation's most sensitive root key (even if it's currently cloud-managed) — who would be the M-of-N custodians, and under what documented conditions would reconstruction ever happen?

Across the series

  • The lifecycle in code: key versions, rotation without re-encrypting data, and crypto-shredding in the Chapter 9 envelope lab. Certificate keys and rotation without downtime: TCP/IP guide, Chapter 56 and HTTPS guide, Chapter 16 (lifetimes falling to 47 days by 2029).

Common mistakes

  • Conflating "rotate the key" with "re-encrypt all the data." They're different operations with very different costs; know which one a given incident or policy actually requires.
  • No meaningful encryption context. Without it, "was this specific tenant/record decrypted" is unanswerable during an incident, forcing worst-case (notify-everyone) scoping.
  • A root/master key one engineer can export or that lives in a password manager. Defeats the entire purpose of an HSM-backed hierarchy.
  • Escrow nobody reviews. A support-access key path or a vendor backdoor is a standing risk until someone asks "who can use this, and how would we know?"
  • Same role administers keys and data. One compromised account, full access. Separate the duties.
  • No plan for full re-encryption. Discovering you can't migrate ciphertext to a new key during an active incident is a bad time to design that pipeline.

Check yourself

  1. Distinguish backing-key rotation from data re-encryption, and say which is usually automatic and cheap.
  2. Why does a layered key hierarchy (root/CMK/DEK) make rotation manageable?
  3. What is a key ceremony and what does M-of-N split knowledge protect against?
  4. Why should key administrators and data administrators be different roles?
  5. In the worked incident, what made it possible to scope the breach precisely instead of notifying every customer?
  6. BYOK vs HYOK — what does each change about who controls the key, and what new dependency does HYOK introduce?

Further reading

  • Standard: NIST SP 800-57 Part 1 (Key Management — general), Part 2 (organizational), Part 3 (application-specific) — the authoritative, exhaustive reference for every stage in this chapter.
  • Reference: the DNSSEC root KSK ceremony documentation (iana.org publishes the procedures and even video recordings) — a real, public example of M-of-N key ceremonies at the highest stakes.
  • Docs: AWS KMS "key rotation", "multi-Region keys", "External Key Store (XKS)"; GCP "Customer-managed encryption keys" and "External Key Manager"; Azure Key Vault "Bring your own key (BYOK)".
  • Book: Cryptography Engineering (Ferguson, Schneier, Kohno) — the key management chapter, still the clearest conceptual treatment available.

Generated from real-life-security-guide-v1.md · 8 Oct 2026 16:36