Chapter 43 — Tokenization, FPE, and PCI scope reduction

How it works

Tokenization models:

   VAULT-BASED (the common, safest default)
     A dedicated TOKEN VAULT stores: token <-> encrypted-original mapping.
     Generating a token: pick a random value (or a counter/UUID), store
     the mapping, return the token. DETOKENIZING requires calling the
     vault's API with its own strict authorization -- most systems never
     get this permission.
     + tokens are provably non-reversible outside the vault (no math
       relates token to value -- a stolen token is worthless anywhere
       else).
     + the vault is a small, hardened, easy-to-audit system -- ONE thing
       to lock down instead of every system that touches the data.
     - the vault itself is now extremely high-value: a single point of
       failure and a single point of compromise for every token it holds.
       It must be the most hardened thing you run (HSM-backed, strict
       network isolation, its own strict access model, Ch 44 practices).

   VAULTLESS / FORMAT-PRESERVING ENCRYPTION (FPE)
     Uses a deterministic, keyed algorithm (NIST SP 800-38G: FF1 or
     FF3-1) that encrypts a value while PRESERVING ITS FORMAT: a 16-digit
     card number encrypts to another valid-looking 16-digit number; an
     SSN to another 9-digit number. No vault lookup needed -- any system
     with the key can tokenize/detokenize algorithmically.
     + fits into legacy schemas/validation with zero schema changes (the
       "token" passes existing length/Luhn-style checks).
     + no vault availability dependency.
     - it IS encryption (reversible with the key) -- for compliance
       purposes this is usually treated as encryption, not full
       tokenization, so scope reduction is weaker unless the KEY itself
       is kept in a separately-scoped, hardened system (which reintroduces
       most of the vault's requirements).
     - deterministic by nature (same input, same output) -- same
       frequency-leak caveat as Ch 42's deterministic encryption.

   Most PCI tokenization products (Stripe, Braintree, a dedicated
   provider, or a self-hosted vault like HashiCorp Vault's "Transform"
   secrets engine in FPE or tokenization mode) offer BOTH modes; pick
   vault-based when scope reduction is the goal.

PCI-DSS scope reduction — the actual payoff:

   WITHOUT TOKENIZATION: every system that stores, processes, or
   transmits the PAN (primary account number) is in the CARDHOLDER DATA
   ENVIRONMENT (CDE) -- full PCI-DSS requirements apply: network
   segmentation, quarterly scans, extensive logging, restricted access,
   annual assessment, etc. For a typical app, that's checkout, order
   history, refunds, support tooling, analytics, backups... a LOT of
   scope.

   WITH TOKENIZATION (using a PCI-validated tokenization provider or a
   properly-scoped vault): the PAN touches ONLY the payment gateway /
   tokenization service (which handles its own PCI compliance) at the
   moment of capture. Every other system -- order history, support,
   analytics, your own database -- stores and uses the TOKEN, which is
   NOT cardholder data. Those systems drop OUT of the CDE.
   SAQ (Self-Assessment Questionnaire) type shrinks accordingly (e.g.
   SAQ A/A-EP for a fully outsourced, tokenized flow, vs the much heavier
   SAQ D for handling PANs directly).

   THE VAULT/GATEWAY ITSELF stays fully in scope, hardened to the whole
   standard -- which is exactly the point: you concentrate the compliance
   burden into ONE small, well-defended system instead of spreading
   "handle real card numbers carefully" across every service that
   touches an order.

   The equivalent logic applies OUTSIDE payments: tokenize SSNs, national
   IDs, or health-record identifiers to shrink the number of systems that
   need HIPAA/state-privacy-law-level controls.