Chapter 44 — Key management lifecycle
In one sentence
A cryptographic key is only as good as its weakest lifecycle stage — generation, distribution, rotation, escrow, and destruction — and getting any one of them wrong (a predictable key, a leaked key, a key nobody can rotate, a backdoor escrow, a key that survives "deletion") undoes every other control in this Part.
Where we are
Part 2 Ch 9 introduced KMS and envelope encryption; Ch 42 used per-tenant keys for field encryption. This chapter is the operational discipline around every key in the system, independent of which cloud or algorithm.
How it works
The lifecycle:
GENERATION
always a CSPRNG (G1 Ch 12), ideally inside an HSM/KMS that never
exports the raw key material for root/master keys. For BYOK
(bring-your-own-key), generate offline with a real entropy source
and import via the provider's secure import ceremony (wrapped with
the provider's public wrapping key -- never transmitted in the
clear).
THE HIERARCHY (recap and extend Part 2 Ch 9):
ROOT / MASTER KEY (HSM-backed, rarely touched, the ultimate trust
anchor -- e.g. your KMS's root of trust, or an offline root CA key
for signing, G1 Ch 29)
-> INTERMEDIATE / CUSTOMER MASTER KEYS (per environment, per
tenant -- these are what you actually reference in policy)
-> DATA ENCRYPTION KEYS (ephemeral, per-object/per-field,
wrapped by the CMK, never stored unwrapped)
Splitting the hierarchy this way means ROTATING the top of the tree
is cheap (re-wrap the layer below; you don't have to touch the
bulk data) while the ROOT changes rarely and under heavy control.
DISTRIBUTION
never transmit a raw key over an unauthenticated or unencrypted
channel. Envelope encryption (Part 2 Ch 9) is itself a distribution
mechanism -- the DEK travels ONLY in its wrapped form; only the KMS
call unwraps it, in memory, for the instant it's needed.
ROTATION -- two different things people conflate:
BACKING-KEY ROTATION ("rotate the CMK"): the KMS creates a NEW key
VERSION under the same key ID; NEW encrypt/wrap operations use the
new version; OLD ciphertext remains decryptable because the KMS
keeps old versions available. Cheap, usually automatable
(annual auto-rotation is a checkbox in most cloud KMS), and does
NOT require touching your bulk data.
DATA RE-ENCRYPTION ("actually re-encrypt everything under a new
key"): required when you suspect the OLD key material itself is
compromised (not just "it's been a year"), or a compliance
standard mandates a hard cryptoperiod. Expensive -- you must
decrypt-then-re-encrypt every object/record. Plan for it (batch
jobs, versioned ciphertext format with a key-id prefix so you can
migrate incrementally) rather than discovering you can't do it
during an incident.
Because DEKs are generated per-object/per-field and wrapped by the
CMK, "rotating the CMK" (cheap) already limits the blast radius of a
CMK compromise going forward -- new data uses the new version
immediately. It's the OLD data, still wrapped by the old version,
that needs the expensive path if that old version is the one that
leaked.
ESCROW AND BACKUP -- deliberately, not accidentally:
Disaster recovery needs SOME way to recover keys if a region/HSM is
lost -- multi-region KMS replication, or an offline backup of a root
key under strict M-of-N control (below). This is legitimate and
necessary.
The RISK version of the same idea is a backdoor: a key escrowed
"for support purposes", a master key one person can export, a
vendor with unilateral key access. Every escrow path must be
justified, access-controlled as tightly as the key itself, and
logged -- treat "can this key be recovered by someone other than an
authorized custodian following a documented procedure" as a
standing audit question.
DESTRUCTION / CRYPTO-SHREDDING
see Part 2 Ch 9 and Ch 46 -- destroying the key that wraps a dataset
makes that dataset permanently unrecoverable, which is how you
satisfy erasure requests and tenant offboarding across every copy
(including backups) simultaneously, PROVIDED no plaintext copy or
unwrapped-key copy exists elsewhere (verify this, don't assume it).
ALGORITHM AGILITY
key metadata should record the ALGORITHM and PARAMETERS it was
created under (G1 Ch 25's crypto-agility principle, applied to keys
specifically) so a future migration (e.g. a post-quantum signing key
rotation) is a planned rollover, not an archaeology project.
HSM operations and key ceremonies (for root/master keys):
FIPS 140-2/3 Level 2 or 3 hardware for root keys -- tamper-evidence
(L2) or tamper-response (L3), so key extraction requires defeating
physical protections, not just software access.
SPLIT KNOWLEDGE / DUAL CONTROL / M-of-N: no single person can
reconstruct a root key alone. A KEY CEREMONY generates the key inside
the HSM (it never exists outside it) and splits any BACKUP/recovery
material into N shares held by different custodians, requiring M of
them (e.g. 3 of 5) to reconstruct -- and reconstruction itself happens
inside a controlled ceremony, logged, witnessed, rare.
(This is exactly how DNSSEC root-zone key ceremonies and most
certificate authority root-key generations work -- publicly documented
processes worth reading once for the model.)
SEPARATION OF DUTIES: whoever administers KEYS (creates, rotates,
sets policy) should not be the same role that administers the DATA
those keys protect, and vice versa. A single compromised admin account
then can't both access the data AND grant itself the key to decrypt
it.
MONITORING: every key USE (encrypt/decrypt/sign) and every key ADMIN
action (create, rotate, policy change, delete) is logged (Part 2 Ch 9's
CloudTrail example) and reviewed; a spike in decrypt calls for one key,
or a policy change outside a change window, is a detection rule
(Ch 54), not just an audit artefact.
BYOK vs HYOK vs cloud-managed:
cloud-managed key simplest; provider generates and holds it.
BYOK (bring your own) you generate the root key material (often in
your own HSM) and import it into the
provider's KMS -- you control generation,
the provider operates the service.
HYOK (hold your own /
external key manager) the key material NEVER enters the
provider's environment; an EXTERNAL KEY
STORE (AWS XKS, GCP External Key Manager)
on YOUR infrastructure services
encrypt/decrypt calls -- maximum control,
maximum operational burden, and a hard
dependency: if your external KMS is down,
the cloud provider cannot decrypt ANYTHING
that depends on it, including their own
support access.
Worked example
A CMK-compromise response, showing why the hierarchy matters.
INCIDENT: an over-permissioned IAM role that had `kms:Decrypt` +
`kms:GenerateDataKey` on the `prod-orders` CMK is discovered to have
been usable by a compromised CI job for the last 30 days (Part 3
Ch 12's OIDC-trust-misconfiguration scenario, realised).
RESPONSE, using the lifecycle:
1. IMMEDIATE: revoke the role's access to the key (update the key
policy); this stops FUTURE encrypt/decrypt calls with that
identity, but does nothing about ciphertext already exposed if
the DEKs were exfiltrated (they generally aren't -- envelope
encryption keeps the DEK's plaintext form transient and in-
process, Part 2 Ch 9 -- but you must verify no logging/caching
captured it).
2. BACKING-KEY ROTATION: rotate `prod-orders` to a new key version
immediately. Cheap, automatic. All NEW data is now wrapped by a
version the compromised role never had a session for (a session's
temporary credentials also expire on their own, Part 2 Ch 5-6).
3. ASSESS: pull CloudTrail for every `Decrypt`/`GenerateDataKey` call
by that role in the 30-day window -- the ENCRYPTION CONTEXT
(Part 2 Ch 9) on each call tells you exactly WHICH tenant's data
was actually decrypted (not just "the key was used" -- which
record). This is only possible because you set a meaningful
encryption context per call.
4. SCOPE: for the specific tenants/records confirmed decrypted
during the compromise window, this is now a data breach requiring
the G1 Ch 51 incident-response and notification process --
scoped precisely because of step 3, not "notify everyone
because we can't tell."
5. DATA RE-ENCRYPTION: NOT required for the whole dataset -- only the
OLD key version is potentially exposed, and only for records
confirmed accessed; if a full re-encryption is still warranted by
policy, you now have the versioned-ciphertext migration path
(above) ready rather than improvised.
6. LONG TERM: fix the IAM/OIDC misconfiguration (Part 3 Ch 12),
require encryption-context-per-call org-wide as a policy, and add
the "decrypt volume by role" detection rule (Ch 54) that should
have caught this in days, not 30.
Practice (60 min)
- In your sandbox, enable automatic annual key rotation on a customer- managed KMS key; read the docs on exactly what rotation does and doesn't re-encrypt.
- Design (on paper) your key hierarchy for SecureShop: root -> per-environment CMKs -> per-tenant CMKs -> per-record DEKs. Write down who/what can administer each layer and who/what can only use each layer (separation of duties).
- Simulate the incident above: revoke a role's key-policy access, rotate the key version, and query your audit log filtered by encryption context to scope exactly what was exposed.
- Research (don't necessarily implement) BYOK for one of your KMS keys — what would change operationally, and what new failure mode (losing your own HSM) would you now own?
- Write a one-page "key ceremony" procedure for your organisation's most sensitive root key (even if it's currently cloud-managed) — who would be the M-of-N custodians, and under what documented conditions would reconstruction ever happen?
Across the series
- The lifecycle in code: key versions, rotation without re-encrypting data, and crypto-shredding in the Chapter 9
envelopelab. Certificate keys and rotation without downtime: TCP/IP guide, Chapter 56 and HTTPS guide, Chapter 16 (lifetimes falling to 47 days by 2029).
Common mistakes
- Conflating "rotate the key" with "re-encrypt all the data." They're different operations with very different costs; know which one a given incident or policy actually requires.
- No meaningful encryption context. Without it, "was this specific tenant/record decrypted" is unanswerable during an incident, forcing worst-case (notify-everyone) scoping.
- A root/master key one engineer can export or that lives in a password manager. Defeats the entire purpose of an HSM-backed hierarchy.
- Escrow nobody reviews. A support-access key path or a vendor backdoor is a standing risk until someone asks "who can use this, and how would we know?"
- Same role administers keys and data. One compromised account, full access. Separate the duties.
- No plan for full re-encryption. Discovering you can't migrate ciphertext to a new key during an active incident is a bad time to design that pipeline.
Check yourself
- Distinguish backing-key rotation from data re-encryption, and say which is usually automatic and cheap.
- Why does a layered key hierarchy (root/CMK/DEK) make rotation manageable?
- What is a key ceremony and what does M-of-N split knowledge protect against?
- Why should key administrators and data administrators be different roles?
- In the worked incident, what made it possible to scope the breach precisely instead of notifying every customer?
- BYOK vs HYOK — what does each change about who controls the key, and what new dependency does HYOK introduce?
Further reading
- Standard: NIST SP 800-57 Part 1 (Key Management — general), Part 2 (organizational), Part 3 (application-specific) — the authoritative, exhaustive reference for every stage in this chapter.
- Reference: the DNSSEC root KSK ceremony documentation (
iana.orgpublishes the procedures and even video recordings) — a real, public example of M-of-N key ceremonies at the highest stakes. - Docs: AWS KMS "key rotation", "multi-Region keys", "External Key Store (XKS)"; GCP "Customer-managed encryption keys" and "External Key Manager"; Azure Key Vault "Bring your own key (BYOK)".
- Book: Cryptography Engineering (Ferguson, Schneier, Kohno) — the key management chapter, still the clearest conceptual treatment available.