Chapter 13 — Securing the software factory: provenance and admission

In one sentence

Close the loop between "reviewed source" and "running workload" by producing signed provenance and an SBOM at build time and verifying them at deploy time, so only artefacts built the approved way from the approved source can run.

Where we are

Chapter 12 hardened the pipeline. But a hardened pipeline can still be bypassed (someone pushes an image straight to the registry, or a build server is compromised — SolarWinds, G1 Ch 47). Provenance + admission verification makes the deploy step check rather than trust.

How it works

SLSA — Supply-chain Levels for Software Artifacts (v1.0 Build track):

   Build L0   no guarantees.
   Build L1   provenance EXISTS -- a machine-readable record of how the
              artefact was built (builder, source, entry point).
   Build L2   provenance is SIGNED and generated by a HOSTED build
              platform (not a laptop). Tampering after the build is
              detectable.
   Build L3   the build platform is HARDENED: builds are isolated from
              each other, secret material for signing is inaccessible to
              user-defined build steps, provenance is non-falsifiable even
              by someone who controls the build definition.
   Aim for L3 for anything that runs in prod. GitHub Actions + the
   `slsa-github-generator` reusable workflow gets you to L3.

Provenance (in-toto attestation / SLSA predicate): a signed statement that says "artefact with digest X was built by builder B from source repo R at commit C, using build config F, with these materials/inputs". It travels with the artefact (as an OCI referrer, or in Rekor).

SBOM (G1 Ch 47): the component inventory. Also produced at build, also signed, also attached.

Signing with Sigstore:

   cosign  -- signs container images and blobs.
   Fulcio  -- a CA that issues SHORT-LIVED (10 min) certs bound to an
              OIDC identity (the CI workflow's identity, or a human's
              Google/GitHub login). "Keyless" -- no long-lived signing key
              to steal.
   Rekor   -- a public (or your private) transparency log of signatures
              and attestations -> tamper-evident, auditable.

   In CI (keyless, identity = the workflow):
     cosign sign --yes  $IMAGE@$DIGEST
     cosign attest --yes --predicate sbom.spdx.json \
        --type spdxjson $IMAGE@$DIGEST
     # SLSA provenance via slsa-github-generator, also attested

Verify at admission (the enforcement point):

   A Kubernetes admission controller checks EVERY pod's images before they
   run:
     - signature is valid AND
     - the signing identity is the expected CI workflow
       (--certificate-identity
        "https://github.com/secureshop/platform/.github/workflows/release.yml@refs/heads/main"
        --certificate-oidc-issuer "https://token.actions.githubusercontent.com")
       AND
     - required attestations are present and valid (SBOM, SLSA provenance
       with source repo = secureshop/*, no CRITICAL vulns in the scan
       attestation) AND
     - the image is from an approved registry AND referenced BY DIGEST.

   Tools: Sigstore `policy-controller`, Kyverno `verifyImages`,
   Connaisseur, Ratify + OPA Gatekeeper, or your cloud's native
   (AWS Signer + EKS, GKE Binary Authorization, Azure Defender).

   RESULT: an image pushed straight to the registry by an attacker -- with
   no valid signature from the release workflow's identity -- is REFUSED
   admission. The build being bypassed no longer means prod is bypassed.

Hermetic / reproducible builds (raises the bar further):

   HERMETIC   the build has NO network access and all inputs are pinned
              and content-addressed -> a poisoned mirror or a
              pulled-at-build-time dependency can't sneak in.
   REPRODUCIBLE  the same inputs always produce a bit-identical output ->
              anyone can rebuild and verify, and two independent builders
              agreeing is strong evidence of integrity.
   Enablers: Bazel, Nix, `ko` (Go), Jib (Java), apko/melange (Chainguard),
   `docker buildx` with pinned digests and `--no-cache`, and running the
   build in a network-egress-denied sandbox.

The paved road (make the secure path the easy path):

   Provide a golden pipeline template that ALREADY does: SBOM, scan,
   sign, provenance, push-by-digest, deploy-via-GitOps-with-verification.
   Teams that use it get security for free. Teams that roll their own get
   flagged by admission (their images won't verify). The incentive points
   the right way -- "psychological acceptability" (G1 Ch 52) as
   architecture.

Worked example

Admission refusing an out-of-band image.

   Kyverno ClusterPolicy (abbreviated):
     verifyImages:
       - imageReferences: ["registry.secureshop.internal/*"]
         failureAction: Enforce
         attestors:
           - entries:
             - keyless:
                 subject: "https://github.com/secureshop/platform/.github/workflows/release.yml@refs/heads/main"
                 issuer: "https://token.actions.githubusercontent.com"
         attestations:
           - type: "https://slsa.dev/provenance/v1"
             conditions:
               - all:
                 - key: "{{ buildDefinition.externalParameters.workflow.repository }}"
                   operator: Equals
                   value: "https://github.com/secureshop/platform"

   Scenario A: `kubectl set image deploy/orders-api
     orders-api=registry.secureshop.internal/orders-api:hotfix` where
     `:hotfix` was `docker push`ed by an engineer from a laptop.
     -> Kyverno: no signature from the release workflow identity ->
        admission DENIED. The engineer must go through the pipeline.

   Scenario B: attacker with registry write pushes
     `registry.secureshop.internal/orders-api:v9.9.9` containing a
     backdoor. A compromised in-cluster process tries to run it.
     -> no valid keyless signature from the expected CI identity ->
        DENIED. Registry write alone is not enough to run in prod.

   Scenario C: the real release workflow builds v2.4.0, signs it, attests
     SBOM + SLSA provenance, pushes by digest, GitOps updates the manifest
     to the digest. -> all checks pass -> admitted. Normal path, secured.

Practice (75 min)

In a repo, add a release job that: builds an image, generates an SBOM (syft), runs grype/trivy, cosign sign (keyless), cosign attest the SBOM, and (bonus) SLSA provenance via slsa-github-generator. Push by digest.
cosign verify and cosign verify-attestation the image locally with --certificate-identity / --certificate-oidc-issuer. Tamper with the image (repush different content, same tag) — watch verification fail.
In a kind cluster, install Kyverno (or Sigstore policy-controller) with a verifyImages policy pinned to your workflow identity.
Deploy the signed image → admitted. docker push an unsigned image and try to run it → denied. Screenshot both.
Add an attestation check: require a SLSA provenance whose source repo matches yours; try an image built from a different repo → denied.
Write the "paved road" pipeline template as a reusable workflow and document it.

Common mistakes

Signing but not verifying. A signature nobody checks is decoration. The value is the admission gate.
Verifying the signature but not the identity. "Is it signed?" — yes, by the attacker. Pin --certificate-identity to the exact workflow.
Allowing images by tag. Tags are mutable; verify and deploy by digest.
No provenance, just a signature. A signature says "someone signed this"; provenance says "built from repo R at commit C by builder B" — that's what you actually want to assert.
Admission policy in Audit/warn mode forever. Roll to Enforce, or attackers just ignore the warning.
Forgetting init containers, sidecars, ephemeral/debug containers, and CronJob pod templates. Verify all image-bearing fields.
A paved road that's harder than rolling your own. Then nobody uses it. Invest in DX.

Check yourself

What do SLSA Build L1, L2, and L3 each add?
What does provenance assert that a bare signature does not?
In keyless signing, what is Fulcio and what is Rekor, and why is "no long-lived key" a security win?
At admission, name four things you should verify about an image.
What does a hermetic build prevent that a normal build does not?
Why does the "paved road" idea improve security rather than just DX?