Chapter 16 — Kubernetes architecture and its trust boundaries

In one sentence

Every request in Kubernetes goes through the API server, all state lives in etcd, and the kubelet on each node holds credentials and runs whatever the API server tells it — so those three components, and the trust edges between them, are where cluster security is won or lost.

Where we are

You can secure a pod (Ch 15). Now the cluster: what the components are, what credentials each holds, and which boundaries an attacker crosses moving from "pod" to "node" to "cluster-admin".

How it works

The control plane:

   kube-apiserver   THE front door. Everything -- kubectl, controllers,
     kubelets, the dashboard -- talks only to it. It does, in order:
       AUTHN     client certs, bearer tokens (SA / OIDC), webhook, (proxy
                 headers). No "anonymous" in a good config
                 (--anonymous-auth=false).
       AUTHZ     Node authorizer + RBAC (+ optionally ABAC/Webhook).
       ADMISSION mutating webhooks -> schema validation -> validating
                 webhooks -> validating admission policies (CEL).
       then persist to etcd.
     Compromise = total.

   etcd            The key-value store holding ALL cluster state: every
     object, and every Secret. Secrets are base64 (NOT encrypted) unless
     you configure EncryptionConfiguration (aescbc/aesgcm/kms provider).
     Anyone who can read etcd (or an etcd backup, or a snapshot) has every
     Secret and can write any object -> cluster-admin equivalent.
     Controls: mTLS (peer + client), encryption at rest with a KMS
     provider, its own isolated network, encrypted + access-controlled
     backups, not co-located with workloads.

   kube-scheduler / kube-controller-manager   control loops with powerful
     credentials (the controller-manager can create tokens, approve CSRs,
     manage most objects). Run on control-plane nodes only.

   cloud-controller-manager   holds cloud credentials (create LBs, disks,
     routes). A privesc bridge to the cloud account if abused.

The node:

   kubelet          The node agent. Talks to the API server with a client
     cert identity `system:node:<nodeName>` (in the `system:nodes` group).
     The NODE AUTHORIZER limits it to objects related to pods on ITS node
     -- but that includes reading the Secrets, ConfigMaps, and SA tokens
     of every pod scheduled there. Its own API (:10250) must require
     auth (--anonymous-auth=false, --authorization-mode=Webhook); an open
     kubelet API historically allowed unauthenticated `exec`/`run` into
     any pod on the node.
   container runtime (containerd / CRI-O)   pulls images, runs containers.
     Its socket = node takeover (Ch 15).
   kube-proxy       programs Service routing (iptables/ipvs/eBPF).
   CNI plugin       pod networking AND NetworkPolicy enforcement -- but
     ONLY if the plugin supports it (Calico, Cilium, Antrea, Weave;
     flannel does not). "We have NetworkPolicies" is meaningless if the
     CNI ignores them.

The trust boundaries (and the moves across them):

   internet -> apiserver        exposed API server + weak authn/authz, or
                                a leaked kubeconfig / SA token.
   pod -> apiserver             the pod's mounted SA token. What can it do?
                                (Ch 17) Also: can the pod even REACH the
                                API server / kubelet / etcd? (NetworkPolicy,
                                Ch 19)
   pod -> node                  the escape classes (Ch 15).
   node -> cluster              the kubelet's credentials + the Node
                                authorizer + reading on-node Secrets (incl.
                                powerful controller SA tokens if such a pod
                                runs there) + static-pod manifests dir.
   node/pod -> cloud            IMDS from a pod with hostNetwork or a
                                permissive metadata setup -> the node's
                                instance role (Part 2). IRSA misconfig
                                (Ch 20).
   any-write-to-etcd -> all     etcd access, an etcd backup, or a
                                snapshot in an over-shared bucket.
   admission webhook control -> all   whoever can create/modify
                                Validating/MutatingWebhookConfigurations
                                sees or rewrites every API request, or
                                wedges the cluster.

Worked example

Pod token to cluster-admin via a secret-reading path.

   START: RCE in a pod in namespace `edge`. Its SA is `edge:web` with
   RBAC: get/list pods and services in `edge` only. Modest.

   MOVE 1  the pod also has (mistakenly) `get`/`list` on `secrets` in
     `kube-system` (someone copy-pasted a ClusterRoleBinding).
       kubectl --token=$(cat /var/run/secrets/.../token) \
         get secrets -n kube-system -o yaml
     -> among them, a legacy Secret-type token for the
        `namespace-controller` or `generic-garbage-collector` SA
        (cluster-wide powers), or a CI SA token stored as a Secret.

   MOVE 2  use that token:
       kubectl --token=$STOLEN auth can-i --list
     -> `*` on `*`. Cluster-admin.

   MOVE 3  persist:
       create a MutatingWebhookConfiguration that injects an attacker
       sidecar + token exfil into every new pod, or a ClusterRoleBinding
       for a new SA, or a static pod on a control-plane node.

   CUTS (independent):
     - No pod should have `secrets` access outside its own namespace, and
       rarely even there -> RBAC review + a policy denying broad secret
       reads (Ch 17/18).
     - No long-lived Secret-type SA tokens -> rely on bound projected
       tokens (1.24+), and a policy/audit that flags `type:
       kubernetes.io/service-account-token` Secrets.
     - Encrypt etcd at rest so a snapshot leak isn't game over.
     - NetworkPolicy so `edge` pods can't even reach the API server unless
       they need to (`automountServiceAccountToken: false` for `web`).
     - Alert on `MutatingWebhookConfiguration` create/update (Ch 54).

Practice (45 min) — kind cluster

Map your cluster: kubectl get --raw /version, kubectl api-resources, kubectl get componentstatuses, kubectl -n kube-system get pods. Identify apiserver, etcd, scheduler, controller-manager, kube-proxy, the CNI.
Check the dangerous settings: is --anonymous-auth false? is the kubelet --authorization-mode Webhook? is etcd encryption at rest configured (kubectl get secrets ... -o json then ETCDCTL_API=3 etcdctl get ... on the control-plane node — is the value plaintext)? Fix in kind by editing the cluster config with an EncryptionConfiguration.
From a pod, try to reach: the API server, the kubelet :10250, etcd :2379, the metadata endpoint. Note which are reachable (they mostly are, by default — Ch 19 fixes that).
kubectl auth can-i --list as the default SA in a namespace. Then as system:node:<node> (--as). Understand the Node authorizer's scope.
Run kubescape scan framework nsa and read the control-plane findings.

Common mistakes

Exposed API server with weak authn. A public endpoint plus anonymous auth, a static token file, or a leaked admin kubeconfig is instant cluster-admin.
etcd unencrypted at rest. A backup in an over-shared bucket = every Secret. Configure a KMS/aesgcm provider and encrypt backups.
Trusting NetworkPolicy with a CNI that ignores it (flannel). Policies silently do nothing.
Legacy Secret-type SA tokens lying around. Long-lived, non-expiring cluster credentials. Migrate to bound tokens; flag the Secrets.
Kubelet API open / anonymous. Historically an unauthenticated exec into any pod on the node.
Not watching admission-webhook config changes. Controlling a webhook = controlling every API request.

Check yourself

What are the three components where cluster security is decided, and why each?
What identity does the kubelet use, and what does the Node authorizer let it do that matters for an attacker?
Why is etcd (or an etcd backup) access equivalent to cluster-admin?
What must a CNI do for NetworkPolicy to mean anything?
Name three trust boundaries an attacker crosses going from pod to cluster-admin.
Why is control over a MutatingWebhookConfiguration so powerful?