Chapter 20 — Runtime security and workload identity

In one sentence

Runtime security watches running containers for behaviour that shouldn't happen (a shell, a new binary, an unexpected connection, a container-escape syscall pattern), and workload identity gives each pod a scoped, short-lived credential to the cloud so you never mount a static key.

Where we are

Prevention (Ch 15-19) reduces what's possible. Runtime detection catches what gets through, and it's your primary signal that a pod is compromised (feeding Ch 54-55). Workload identity closes the "pod needs to call S3/KMS/Secrets Manager" gap without a leakable credential.

How it works

Runtime detection tools:

   FALCO (CNCF)     syscall + k8s-audit + container events, via a kernel
     module or a (preferred) eBPF probe. YAML rules. Ships a solid default
     ruleset: "shell spawned in container", "read sensitive file
     untrusted", "write below /etc", "unexpected outbound", "container
     drift" (a binary not in the image runs), "escape attempt" syscall
     patterns, "contact k8s API server from container". Falcosidekick fans
     alerts out (Slack, SIEM, EventBridge...).
   TETRAGON (Cilium)  eBPF, TracingPolicy CRDs. Not just observe --
     can ENFORCE (SIGKILL a process, block a syscall) in kernel. Rich
     process ancestry. Lower overhead at high event rates.
   TRACEE (Aqua)    eBPF, behavioural detection "signatures", good for
     forensics/replay.
   Also: cloud-native (GuardDuty EKS Runtime Monitoring, Defender for
     Containers), and commercial (Sysdig, Aqua, Wiz Runtime).

   Tune aggressively -- the default rules are noisy in a real cluster.
   Baseline your workloads, allowlist known-good, and route only
   high-fidelity rules to paging.

Complementary preventive runtime controls (from Ch 15, applied):

   seccompProfile: RuntimeDefault (or a custom profile from
     `strace`/`oci-seccomp-bpf-hook`/inspektor-gadget). Blocks the
     syscalls escapes need.
   AppArmor/SELinux profile per workload.
   readOnlyRootFilesystem + no package manager in the image -> "container
     drift" (attacker installs tools) becomes impossible / loud.
   Distroless images -> no shell for the attacker to spawn (Falco's
     "shell in container" becomes "there is no shell", and any exec is
     anomalous).

Workload identity (no static cloud keys in pods):

   AWS IRSA (IAM Roles for Service Accounts)
     - the cluster has an OIDC provider (its issuer URL registered in IAM).
     - a mutating webhook injects a PROJECTED SA TOKEN (audience
       sts.amazonaws.com) + env vars into pods whose SA is annotated
       `eks.amazonaws.com/role-arn`.
     - the SDK calls sts:AssumeRoleWithWebIdentity; the role's TRUST
       POLICY conditions on
         "<oidc>:sub": "system:serviceaccount:<ns>:<sa>"
         "<oidc>:aud": "sts.amazonaws.com"
       -> short-lived creds scoped to that role.
     MISCONFIG: trust `:sub` set to `system:serviceaccount:*:*` or a
       wildcard namespace -> any pod assumes the role. Pin ns AND sa.
   AWS EKS Pod Identity (newer)  -- an agent + a PodIdentityAssociation
     API; no per-cluster OIDC federation to manage; role trust is on
     `pods.eks.amazonaws.com`. Simpler; prefer for new clusters.
   GCP Workload Identity  -- bind a KSA to a GSA:
     `gcloud iam service-accounts add-iam-policy-binding GSA
        --role roles/iam.workloadIdentityUser
        --member "serviceAccount:PROJECT.svc.id.goog[NS/KSA]"`
     and annotate the KSA. The GKE metadata server brokers tokens; the
     legacy node metadata paths are blocked.
   Azure Workload Identity  -- a federated credential on a user-assigned
     managed identity, trusting the cluster's OIDC issuer + the KSA
     subject; a webhook injects the token.

   RULES:
     [ ] one role per workload, least privilege (Part 2 Ch 6).
     [ ] trust conditions pin the EXACT namespace + serviceaccount (+
         audience). Review every IRSA trust policy for wildcards.
     [ ] `automountServiceAccountToken: true` is required for IRSA (the
         projected token) -- but the token's audience is STS, not the k8s
         API; still set `automount: false` for the DEFAULT SA and only
         enable per-workload.
     [ ] node instance role: minimal (CNI, ECR pull, CloudWatch) -- pods
         must NOT be able to reach it (Ch 19 metadata block) so they can't
         borrow the node's (broader) permissions.

Worked example

A Falco rule catching a container escape attempt, wired to response.

   Custom Falco rule (on top of the defaults):
     - rule: Escape via release_agent or nsenter
       desc: process attempts a known container-escape technique
       condition: >
         spawned_process and container and
         ((proc.name = nsenter and proc.args contains "-t 1") or
          (proc.cmdline contains "release_agent") or
          (proc.name in (mount) and proc.args contains "cgroup") or
          (fd.name startswith /proc/1/root))
       output: >
         Container escape attempt (pod=%k8s.pod.name ns=%k8s.ns.name
         image=%container.image.repository proc=%proc.cmdline
         user=%user.name)
       priority: CRITICAL
       tags: [container, escape, mitre_privilege_escalation, T1611]

   FLOW:
     Falco -> Falcosidekick -> (a) page the on-call, (b) an EventBridge/
       webhook that calls a response function:
         - cordon the node, evict other pods
         - delete the offending pod, set the Deployment replicas to 0
         - snapshot the node's disk + capture the pod's memory (Ch 55)
         - revoke the workload's IRSA role session (deny-list its role /
           rotate) and the node instance role
         - open an incident with the Falco output attached
   Tetragon alternative: a TracingPolicy that SIGKILLs the process on the
   same condition -- prevention, not just detection -- while the response
   function handles containment.

Practice (60 min) — kind cluster

Install Falco (eBPF driver) + Falcosidekick. Trigger the defaults: kubectl exec a shell into a pod, cat /etc/shadow, curl an external IP, install a package. Watch the alerts; note fidelity.
Write two custom rules: the escape-attempt rule above, and "process not in the image's expected set for orders-api". Test both.
Swap a workload to a distroless image; try kubectl exec -- sh (no shell); confirm any exec now trips an anomaly rule.
Install Tetragon; write a TracingPolicy that kills any process matching the escape condition; verify it actually SIGKILLs.
Set up IRSA (or, on kind, simulate with Minikube + LocalStack + a projected token, or do it for real in an EKS sandbox): one SA, one least-priv role, trust pinned to ns:sa. Prove a pod with a different SA cannot assume the role.
Audit every IRSA trust policy in a real account for wildcard sub.

Across the series

A least-privilege Go workload end to end: distroless non-root image (14.8 MB, no shell), readOnlyRootFilesystem, dropped capabilities, seccomp RuntimeDefault, and a hardened systemd unit scored 3.1 by systemd-analyze security: Linux guide, Chapter 82.
Workload identity in code: Chapter 23's SPIFFE lab, in this guide.

Common mistakes

Falco/Tetragon deployed, rules untuned, alerts ignored. Noise = blind. Baseline, allowlist, and only page on high-fidelity rules.
Detection without response wiring. An escape alert at 3am with no runbook or automation is a slow incident.
IRSA trust policies with wildcard namespace or SA. Any pod in the cluster assumes the role. Pin ns:sa and the audience.
Pods able to reach the node metadata endpoint. They borrow the node instance role, which is usually broader than any pod role. Block it (Ch 19).
Giving the node instance role app permissions "to keep it simple". Every pod on the node inherits them via IMDS. Keep node role minimal; use workload identity for apps.
Images with a shell + package manager in prod. Removes your best anomaly signal and gives the attacker tools. Go distroless / read-only.

Check yourself

Name three things a good runtime ruleset detects, and why distroless + read-only rootfs makes detection easier.
Falco vs Tetragon — the key capability difference.
Walk through how IRSA gives a pod short-lived AWS creds, and the one trust condition you must pin.
Why must pods be unable to reach the node metadata endpoint even with workload identity configured?
What should the node instance role contain, and what should it not?
Why wire runtime alerts to automated containment, and what might that containment do?