Frontend Labs
⋯
Read as one page Markdown source ↗

Part VI — Security

34. Securing the cluster and the workloads

Defense in depth, in the order an attacker meets it: supply chain (§8, §34.6) → admission (what is allowed to run) → runtime privileges (what a compromised pod can do) → network (what it can reach) → identity (what it can access) → detection (Falco, audit logs).

34.1 ServiceAccounts and RBAC

Every workload gets its own ServiceAccount. Mount a token only if it calls the Kubernetes API.

apiVersion: v1
kind: ServiceAccount
metadata: { name: orders, namespace: orbit }
automountServiceAccountToken: false

RBAC: Role/ClusterRole (verbs on resources) + RoleBinding/ClusterRoleBinding (to users, groups, ServiceAccounts). Example: a read-only on-call role for the orbit namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: oncall-readonly, namespace: orbit }
rules:
  - apiGroups: ["", "apps", "batch", "autoscaling", "keda.sh", "argoproj.io"]
    resources: ["pods", "pods/log", "services", "endpoints", "configmaps", "events",
                "deployments", "replicasets", "statefulsets", "jobs", "cronjobs",
                "horizontalpodautoscalers", "scaledobjects", "rollouts"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/portforward", "pods/ephemeralcontainers"]   # debugging, audited
    verbs: ["create", "patch"]
  # deliberately NO secrets
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: oncall-readonly, namespace: orbit }
subjects: [{ kind: Group, name: "orbit-oncall", apiGroup: rbac.authorization.k8s.io }]   # from your SSO/OIDC
roleRef: { kind: Role, name: oncall-readonly, apiGroup: rbac.authorization.k8s.io }

kubectl auth can-i list secrets -n orbit --as-group=orbit-oncall --as=jane
kubectl auth can-i --list -n orbit --as=system:serviceaccount:orbit:orders

RBAC escalation traps: create pods in a namespace means access to every Secret and ServiceAccount there (mount it into a new pod). escalate, bind and impersonate verbs, wildcards (*), and nodes/proxy are effectively cluster-admin. Audit them with tools like rbac-tool or kubectl-who-can.

34.2 Pod Security Admission

Built in (replaced PodSecurityPolicy). Label namespaces with a level: privileged, baseline, or restricted.

kubectl label namespace orbit \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

restricted requires exactly the securityContext in our golden manifest: non-root, no privilege escalation, drop: ALL, a seccomp profile. Roll it out with warn/audit first, read the warnings, then enforce.

34.3 NetworkPolicy: default deny, then allow what's needed

By default every pod can talk to every pod in every namespace. A compromised catalog could reach Postgres, the Kubernetes API, and the cloud metadata endpoint. NetworkPolicies need a CNI that enforces them (Cilium, Calico; recent kind's kindnet does).

# 1. default deny everything in the namespace (ingress AND egress)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: orbit }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# 2. everyone may resolve DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: orbit }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
          podSelector: { matchLabels: { k8s-app: kube-dns } }
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
---
# 3. the gateway may reach catalog and orders
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-from-gateway, namespace: orbit }
spec:
  podSelector:
    matchExpressions: [{ key: app.kubernetes.io/name, operator: In, values: [catalog, orders] }]
  policyTypes: [Ingress]
  ingress:
    - from: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: envoy-gateway-system } } }]
      ports: [{ port: 8080 }]
---
# 4. Postgres accepts only orders, catalog, worker (+ CNPG operator and its own replicas)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: postgres-ingress, namespace: orbit }
spec:
  podSelector: { matchLabels: { cnpg.io/cluster: orbit-db } }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchExpressions: [{ key: app.kubernetes.io/name, operator: In, values: [catalog, orders, worker] }]
        - podSelector: { matchLabels: { cnpg.io/cluster: orbit-db } }                   # replication
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: cnpg-system } }  # operator
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: monitoring } }   # metrics :9187
      ports: [{ port: 5432 }, { port: 8000 }, { port: 9187 }]
---
# 5. app egress: Postgres, Redis, and the node-local OTel agent
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: app-egress, namespace: orbit }
spec:
  podSelector:
    matchExpressions: [{ key: app.kubernetes.io/name, operator: In, values: [catalog, orders, worker] }]
  policyTypes: [Egress]
  egress:
    - to: [{ podSelector: { matchLabels: { cnpg.io/cluster: orbit-db } } }]
      ports: [{ port: 5432 }]
    - to: [{ podSelector: { matchLabels: { app.kubernetes.io/name: redis } } }]
      ports: [{ port: 6379 }]
    - to: [{ ipBlock: { cidr: 172.18.0.0/16 } }]   # node IPs (kind's docker network): OTel agent hostPort
      ports: [{ port: 4317 }]

Write the policies (and the remaining Redis ingress / monitoring policies) while you build the service, not afterwards. Then test them:

kubectl run -n orbit -it --rm probe --image=nicolaka/netshoot --restart=Never -- \
  nc -zv -w2 orbit-db-rw 5432      # expect: timeout (probe pod isn't allowed)

Cilium adds L7 (HTTP method/path) and FQDN egress policies (toFQDNs: api.stripe.com). It also adds Hubble, which shows every dropped flow and why, and turns NetworkPolicy debugging from guesswork into a query.

War story: pods on cloud VMs can reach the instance metadata endpoint (169.254.169.254) by default and fetch the node's IAM credentials. Several cloud breaches started with an SSRF in a web app that fetched that URL. Block it with an egress policy (allow 0.0.0.0/0 except 169.254.169.254/32), enforce IMDSv2 with hop limit 1 on AWS, and use workload identity (§42.3) so pods never need node credentials.

34.4 Secrets management: External Secrets Operator

The source of truth lives in a real secret manager (AWS Secrets Manager, GCP Secret Manager, Vault, Azure Key Vault). ESO syncs it into Kubernetes Secrets and refreshes on rotation. Git holds only references.

helm repo add external-secrets https://charts.external-secrets.io
helm upgrade --install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: { name: aws-secrets-manager }
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:   # the ESO pod's own IAM identity (Pod Identity / IRSA): no static keys
        jwt: { serviceAccountRef: { name: external-secrets, namespace: external-secrets } }
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: payment-provider, namespace: orbit }
spec:
  refreshInterval: 1h
  secretStoreRef: { kind: ClusterSecretStore, name: aws-secrets-manager }
  target: { name: payment-provider, creationPolicy: Owner }
  data:
    - secretKey: API_KEY
      remoteRef: { key: prod/orbit/payment-provider, property: api_key }

Alternatives: Sealed Secrets (encrypt with the cluster's public key, commit the ciphertext; simple, cluster-bound), and SOPS + age/KMS (encrypted YAML in Git, decrypted by Argo CD/Flux plugins). For database credentials, CNPG already generates and stores them.

⚠️ Caveat (verify): (1) NetworkPolicy enforcement on kind depends on the kind/kindnet version; recent releases enforce it, older ones silently ignore policies. Test with the nc probe before trusting the policies, or install Cilium. (2) 172.18.0.0/16 is kind's usual Docker network; confirm with docker network inspect kind | jq '.[0].IPAM'. (3) Check CNPG's instance ports (5432, status 8000, metrics 9187) in its docs. (4) External Secrets uses the external-secrets.io/v1 API on recent releases (older ones use v1beta1), and the AWS auth.jwt shape differs from Pod Identity setups; check the provider docs.

34.5 Admission policy: guardrails as code

ValidatingAdmissionPolicy (built in, CEL expressions, GA since 1.30) needs no extra components:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: orbit-workload-standards }
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - { apiGroups: ["apps"], apiVersions: ["v1"], operations: ["CREATE", "UPDATE"], resources: ["deployments"] }
  validations:
    - expression: "object.spec.template.spec.containers.all(c, !c.image.endsWith(':latest') && c.image.contains(':') || c.image.contains('@sha256:'))"
      message: "images must be pinned to a tag or digest, never :latest"
    - expression: "object.spec.template.spec.containers.all(c, has(c.resources.requests) && has(c.resources.limits) && has(c.resources.limits.memory))"
      message: "every container needs requests and a memory limit"
    - expression: "object.spec.template.spec.containers.all(c, has(c.readinessProbe))"
      message: "every container needs a readinessProbe"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: orbit-workload-standards }
spec:
  policyName: orbit-workload-standards
  validationActions: [Deny]           # start with [Warn, Audit]
  matchResources:
    namespaceSelector: { matchLabels: { orbit.acme.example/enforce: "true" } }

Kyverno (YAML policies; validate, mutate, generate, verify images) or OPA Gatekeeper (Rego) cover what CEL can't, plus a policy library and reporting.

War story: an admission webhook with failurePolicy: Fail whose pods were themselves down blocked every pod creation in the cluster, including its own replacement pods and CoreDNS. Exclude kube-system and the policy engine's own namespace, run ≥ 3 replicas with a PDB, and set webhook timeouts low.

34.6 Only run signed images (Kyverno verifyImages)

CI signs images keylessly (§35). This policy rejects any orbit image not signed by our workflow on our repo:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-orbit-images }
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 15
  rules:
    - name: verify-signature
      match: { any: [{ resources: { kinds: [Pod], namespaces: ["orbit*"] } }] }
      verifyImages:
        - imageReferences: ["ghcr.io/acme/orbit-*"]
          mutateDigest: true                 # rewrite tag -> digest: what was verified is what runs
          attestors:
            - entries:
                - keyless:
                    issuer: https://token.actions.githubusercontent.com
                    subject: "https://github.com/acme/orbit/.github/workflows/ci.yaml@refs/heads/main"
                    rekor: { url: https://rekor.sigstore.dev }

⚠️ Caveat (verify): Kyverno is moving image verification to newer policy types (ImageValidatingPolicy), and spec-level validationFailureAction is deprecated in favor of per-rule failureAction. Check this ClusterPolicy against your Kyverno version with kyverno apply / kubectl apply --dry-run=server. The keyless subject must match your workflow path and ref exactly; check it with cosign verify --certificate-identity ... --certificate-oidc-issuer ... first.

34.7 Runtime detection and continuous audit

  • Falco (eBPF) alerts on suspicious runtime behavior: a shell spawned in a container, writes to /etc, unexpected outbound connections, reading /proc/*/environ.
  • Kubernetes audit logs: who did what via the API. Enable them on managed clusters, ship them to your SIEM, and alert on exec into prod pods, Secret reads by humans, and RBAC changes.
  • trivy-operator / Kubescape: continuously scan running workloads for CVEs, misconfigurations, and exposed secrets.
  • kube-bench: CIS Kubernetes Benchmark checks for nodes and the control plane.

Generated from real-life-k8s-guide.md · 8 Oct 2026 16:36