40. Chaos engineering

Chaos engineering is experiments that test a hypothesis about resilience. It isn't random breakage. Format: "Given steady state X (SLO met), when Y fails, we expect Z, with users seeing at most W."

40.1 Chaos Mesh

helm repo add chaos-mesh https://charts.chaos-mesh.org
helm upgrade --install chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh --create-namespace \
  --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock

# H1: killing 1/3 of orders pods every 2 minutes causes no failed requests (PDB, readiness, retries)
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata: { name: orders-pod-kill, namespace: orbit }
spec:
  schedule: "@every 2m"
  type: PodChaos
  historyLimit: 5
  concurrencyPolicy: Forbid
  podChaos:
    action: pod-kill
    mode: fixed-percent
    value: "33"
    selector: { namespaces: [orbit], labelSelectors: { app.kubernetes.io/name: orders } }
---
# H2: 200ms extra latency to Postgres keeps orders p95 under 300 ms? (tests timeouts/pool sizing)
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata: { name: db-latency, namespace: orbit }
spec:
  action: delay
  mode: all
  selector: { namespaces: [orbit], labelSelectors: { app.kubernetes.io/name: orders } }
  direction: to
  target:
    mode: all
    selector: { namespaces: [orbit], labelSelectors: { cnpg.io/cluster: orbit-db } }
  delay: { latency: "200ms", jitter: "50ms" }
  duration: "10m"
---
# H3: worker memory pressure -> OOMKill -> unacked messages are reclaimed, nothing lost
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata: { name: worker-mem, namespace: orbit }
spec:
  mode: one
  selector: { namespaces: [orbit], labelSelectors: { app.kubernetes.io/name: worker } }
  stressors: { memory: { workers: 1, size: "300MB" } }
  duration: "5m"

⚠️ Caveat (verify): the Chaos Mesh daemon's containerd socket path is correct for kind nodes. Other distros differ (k3s: /run/k3s/containerd/containerd.sock). Check field names against the installed CRDs with kubectl explain podchaos.spec.

40.2 Game day template

Step	Example
Hypothesis	"Losing zone-b causes < 1 min of elevated errors and no data loss"
Steady state	k6 at 200 RPS, SLO dashboards green
Blast radius / abort criteria	staging only; abort if error ratio > 5% for 2 min
Inject	`kubectl cordon` + delete all pods on zone-b nodes (or `docker stop` the kind node)
Observe	alerts fired? which? dashboards tell the story? runbook accurate?
Results	what broke that we didn't expect (there is always something)
Actions	tickets with owners; re-run the experiment after fixes

Real-world example: Netflix's Chaos Monkey (2011) randomly terminated production instances during business hours, so engineers had to build services that tolerated it. The lasting lesson isn't "break prod". It's that failure handling which is never exercised doesn't work when you need it. Start in staging, during work hours, with an abort button.