Chapter 49 — Secrets management

Where secrets leak (in rough order of how often it actually happens)

   1. SOURCE CONTROL      hard-coded in code, committed .env, a config file
                          with real values, a test fixture. Git keeps
                          history forever -- a deleted secret is still in
                          the log, in forks, in clones, on CI caches.
   2. CI/CD LOGS          `echo $API_KEY` for debugging; a tool that prints
                          its config; a stack trace with a connection string.
   3. CONTAINER IMAGES    ARG/ENV secrets baked into a layer; a COPY of the
                          whole repo including .env; `docker history` shows
                          build args.
   4. CLIENT-SIDE         API keys in a mobile app binary or SPA bundle
                          ("it's obfuscated" -- it's `strings` away).
   5. LOGS & APM          request bodies, headers (Authorization), query
                          strings with tokens, error objects.
   6. THIRD PARTIES       Slack messages, Jira tickets, screenshots, wikis,
                          a paste in a support chat.
   7. BACKUPS & SNAPSHOTS a DB dump or VM snapshot with secrets in a table
                          or on disk, stored less carefully than prod.
   8. ENV VARS            readable via /proc, crash dumps, subprocess
                          inheritance, some PaaS dashboards, `ps e`.

The hierarchy of "less bad"

   WORST   hard-coded in source
     |     plaintext config file next to the code
     |     environment variable (better: at least not in git -- but see above)
     |     encrypted-at-rest file, decrypted at deploy (SOPS, git-crypt,
     |         sealed-secrets) -- key management just moved, not solved
     |     a SECRETS MANAGER: Vault, AWS/GCP/Azure Secret Manager,
     |         Kubernetes Secrets + a CSI driver, Doppler, Infisical
   BEST    DYNAMIC, SHORT-LIVED credentials minted per-workload on demand
             (Vault dynamic DB creds, cloud IAM roles / workload identity,
              SPIFFE SVIDs, OIDC-federated CI with no stored secret at all)

The end state to aim for: the application authenticates with a workload identity it didn't have to be given (instance role, K8s ServiceAccount token, OIDC), and exchanges it for short-lived, scoped, audited credentials.

A secrets manager: what it buys you

   * central store, encrypted at rest, TLS in transit
   * fine-grained access policy (this service reads THIS path only)
   * full audit log (who read what, when)
   * versioning + rollback
   * DYNAMIC secrets: Vault creates a DB user with a 1-hour TTL when the
     app asks, and deletes it after -> a leaked cred is useless in an hour
   * automatic ROTATION (static secrets rotated on a schedule / on demand)
   * leasing + revocation: kill all credentials issued to a compromised
     service in one command

Rotation

   Every static secret needs: an owner, an expiry, and a tested rotation
   runbook. If you can't rotate a key in under an hour without an outage,
   that's an incident waiting to happen.

   PATTERN for zero-downtime rotation (works for API keys, signing keys,
   HMAC secrets, DB creds):
     1. Create the NEW secret alongside the old (both valid).
     2. Deploy consumers to ACCEPT both (verify against old OR new).
     3. Deploy producers to USE the new.
     4. Confirm no traffic uses the old (metrics/logs).
     5. Revoke the old.
   Keep 2 key versions live at all times -> rotation is routine, not scary.

Detecting leaked secrets

   PRE-COMMIT / PRE-PUSH   gitleaks, trufflehog, detect-secrets as a git
                           hook -> stop it before it's committed.
   CI SCAN                 same tools on every PR + full-history scan
                           periodically (secrets predate the policy).
   PLATFORM                GitHub/GitLab secret scanning + push protection;
                           partner alerts (many providers auto-revoke on a
                           public leak -- e.g. AWS, Stripe, GitHub tokens).
   MONITORING              alert on use of a secret from an unexpected IP/
                           ASN/time; canary tokens (a fake AWS key that
                           pages you the instant anyone uses it --
                           canarytokens.org).
   IF ONE LEAKS: assume compromised. ROTATE first, investigate second.
   Removing the commit is NOT remediation -- the value must be invalidated.

cd ~/sec-lab/scratch

# 1. Scan a repo (yours) for secrets, including history
docker run --rm -v "$PWD":/repo zricethezav/gitleaks:latest detect \
   --source=/repo --report-format=json --report-path=/repo/gitleaks.json -v
# or: pipx run trufflehog git file://$PWD --only-verified

# 2. Add push protection locally
pipx install detect-secrets
detect-secrets scan > .secrets.baseline
cat > .git/hooks/pre-commit <<'EOF'
#!/bin/sh
detect-secrets-hook --baseline .secrets.baseline $(git diff --cached --name-only) || {
  echo "Potential secret detected. Aborting commit."; exit 1; }
EOF
chmod +x .git/hooks/pre-commit
echo 'aws_secret_access_key = "AKIA................"' > leaky.txt
git add leaky.txt && git commit -m test        # should be BLOCKED

# 3. Run Vault in dev mode and issue a DYNAMIC, short-lived secret
docker run --rm -d --name vault -p 8200:8200 -e VAULT_DEV_ROOT_TOKEN_ID=root hashicorp/vault
export VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN=root
vault secrets enable -path=secret kv-v2
vault kv put secret/secureshop/db username=app password="$(openssl rand -hex 16)"
vault kv get secret/secureshop/db
vault policy write app - <<'EOF'
path "secret/data/secureshop/*" { capabilities = ["read"] }
EOF
vault token create -policy=app -ttl=15m           # scoped, expiring token
# (Full dynamic-DB-creds demo: enable the `database` secrets engine against
#  a throwaway postgres, then `vault read database/creds/app-role` -> a
#  brand-new DB user that self-destructs.)
docker rm -f vault

# 4. Prove env vars leak
KEY=supersecret sh -c 'cat /proc/self/environ | tr "\0" "\n" | grep KEY'   # Linux
KEY=supersecret sh -c 'ps ew $$ | grep -o KEY=supersecret'

Mini-project (45 min): SecureShop secrets baseline

   1. Add gitleaks + detect-secrets to CI; add push protection; run a
      full-history scan and triage/rotate anything found.
   2. Move all SecureShop secrets (DB creds, JWT signing key, third-party
      API keys) into a secrets manager (Vault dev, or your cloud's).
      App reads them at startup via a workload identity, not a committed
      file.
   3. Write and TEST a rotation runbook for the JWT signing key using the
      2-versions-live pattern (verifier accepts old+new; issuer switches;
      retire old). Time yourself -- target < 1 hour.
   4. Drop a canary AWS key (canarytokens.org) into an old-looking
      config path; document what happens if it's "used".
   5. Report -> ~/sec-lab/reports/part11-secrets.md.