33. The Kubernetes debugging playbook

33.1 The universal triage sequence

Whatever the symptom, run these in order. Ninety percent of incidents are diagnosed by step 3.

# 1. What state is it in?
kubectl get pods -n orbit -o wide
# 2. What did Kubernetes try, and what went wrong? (read EVENTS at the bottom)
kubectl describe pod <pod> -n orbit
# 3. What did the app say, including the PREVIOUS (crashed) container?
kubectl logs <pod> -n orbit --previous --tail=200
# 4. What does the controller think? (spec vs status, conditions)
kubectl get deploy orders -n orbit -o yaml | yq '.status'
# 5. What happened around it? (events expire after ~1h by default; ship them to Loki for history)
kubectl get events -n orbit --sort-by=.lastTimestamp | tail -40
# 6. What do the metrics say? (Grafana: restarts, memory vs limit, throttling, RED)
kubectl top pods -n orbit --containers

Best practice: mitigate first, debug second. If a deploy just went out and errors rose, roll back (§36.5) then investigate. Users don't care about your root cause while it's broken. Capture evidence before you mitigate if it's cheap: kubectl logs --previous > file, a heap profile, describe output.

33.2 Symptom → cause → fix

Pending

kubectl describe pod <pod> | sed -n '/Events/,$p'
#   0/4 nodes are available: 3 Insufficient cpu      -> requests too big or cluster full (autoscaler? quota?)
#   ... had untolerated taint {node-role...}         -> needs a toleration / wrong nodeSelector
#   ... didn't match pod affinity/anti-affinity      -> required anti-affinity with too few nodes/zones
#   ... pod has unbound immediate PersistentVolumeClaims -> PVC Pending: check StorageClass/CSI
#   exceeded quota                                   -> ResourceQuota (shows as a ReplicaSet event, not a pod event!)
kubectl get events -n orbit --field-selector reason=FailedCreate     # quota/admission rejections live here

ImagePullBackOff / ErrImagePull

kubectl describe pod <pod> | grep -A3 -i 'failed to pull'
#   not found                    -> typo or tag not pushed (did CI push? which registry?)
#   unauthorized / 403           -> imagePullSecret missing/wrong namespace, or node IAM role
#   no match for platform        -> arm64-only image on amd64 nodes (build multi-arch)
#   toomanyrequests              -> Docker Hub rate limit: mirror images to your own registry

CrashLoopBackOff

kubectl logs <pod> --previous                                  # the crash reason, nearly always
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' | jq
#   exitCode 1  + config error in logs  -> missing env/secret/bad flag: fix config
#   exitCode 137 + reason OOMKilled      -> raise memory limit / fix leak / set GOMEMLIMIT (§23.4)
#   exitCode 137 without OOMKilled       -> killed by liveness probe failing: check probe events
#   exitCode 0                           -> process exited "successfully": wrong command for a server
#   no logs at all                       -> exec format error (arch), missing binary, or crash before logging

Liveness kills look like this in the events: Liveness probe failed: ... Container orders failed liveness probe, will be restarted.

CreateContainerConfigError: a referenced Secret/ConfigMap or key does not exist (describe names it). Common in GitOps when the ExternalSecret hasn't synced yet.

Running but 0/1 READY

kubectl describe pod <pod> | grep -i readiness
kubectl port-forward pod/<pod> 8080:8080 & curl -i localhost:8080/readyz   # what does the app say?

Our /readyz returns JSON listing the failing dependency.

Service returns connection refused / no response

kubectl get endpointslices -n orbit -l kubernetes.io/service-name=orders
#   no endpoints -> selector doesn't match pod labels, OR no pod is Ready
kubectl get svc orders -o yaml | yq '.spec.selector'; kubectl get pods -l app.kubernetes.io/name=orders --show-labels
#   endpoints exist -> port mismatch? targetPort must match containerPort (name or number)
#   still failing -> a NetworkPolicy may be dropping it (§34.3)

DNS failures (no such host, intermittent 5 s delays)

kubectl run -it --rm dns --image=nicolaka/netshoot --restart=Never -- bash
#   dig orders.orbit.svc.cluster.local     (works?)  dig +search orders
#   cat /etc/resolv.conf                    (ndots, search domains)
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50
kubectl -n kube-system top pods -l k8s-app=kube-dns       # CoreDNS saturated? scale it / NodeLocal DNSCache

Stuck Terminating

kubectl get pod <pod> -o jsonpath='{.metadata.finalizers}'      # a controller must remove these
kubectl get node <node>                                         # node NotReady? kubelet can't confirm the kill
# last resort for a pod on a DEAD node (understand the consequences for StatefulSets first):
kubectl delete pod <pod> --grace-period=0 --force

Namespaces stuck Terminating are almost always a CRD-backed resource whose controller was uninstalled first, leaving finalizers that nothing will ever remove. Uninstall in reverse order: CRs, then the operator.

Latency spikes with low average CPU → CPU throttling (§23.3). Check the throttled-periods ratio; remove or raise the CPU limit.

Errors only during deploys → the termination race (§13.5): add a preStop sleep, confirm readiness flips to false on SIGTERM, and set maxUnavailable: 0.

Node NotReady

kubectl describe node <node> | sed -n '/Conditions/,/Addresses/p'   # MemoryPressure/DiskPressure/PIDPressure
kubectl debug node/<node> -it --image=ubuntu                       # shell on the node (host fs at /host)
#   chroot /host journalctl -u kubelet --since "30 min ago" | tail -100
#   chroot /host crictl ps -a ; crictl logs <id>                   # containers seen by the runtime
#   df -h /host/var/lib/containerd                                 # disk full from images/logs?

33.3 Ephemeral debug containers (distroless-friendly)

You can't exec into a distroless pod: there's no shell. Attach a temporary toolbox container into the running pod, sharing its process namespace with --target:

kubectl debug -it orders-7d9f-x2k -n orbit --image=nicolaka/netshoot --target=orders
#   ps aux                       -> you see the app's processes
#   ls -l /proc/1/root/          -> the app container's filesystem
#   ss -tnp                      -> its connections (to DB? stuck in CLOSE_WAIT?)
#   curl -s localhost:8080/readyz
#   tcpdump -i any -nn port 5432 -c 50

# Profiles add capabilities: --profile=netadmin (tcpdump/iptables), sysadmin, general, restricted
kubectl debug -it orders-7d9f-x2k --image=nicolaka/netshoot --target=orders --profile=netadmin

# Copy a crash-looping pod with a different command, to poke at it without the app dying:
kubectl debug orders-7d9f-x2k -it --copy-to=orders-debug --container=orders \
  --image=gcr.io/distroless/static-debian12:debug-nonroot -- sh
kubectl delete pod orders-debug    # clean up

Real-world example: "We need a shell in the image for debugging" is the most common pushback against distroless. Ephemeral containers remove it: tools are attached on demand, by people with RBAC permission for the pods/ephemeralcontainers subresource, and the attach is audited.

33.4 Profiling a live pod

Expose pprof on a separate, non-public port bound to localhost, or behind auth:

// cmd/orders/main.go
import _ "net/http/pprof" // registers on http.DefaultServeMux

go func() { _ = http.ListenAndServe("127.0.0.1:6060", nil) }() // NOT routed by the Service

kubectl port-forward pod/orders-7d9f-x2k 6060:6060 -n orbit
go tool pprof -http=:9999 'http://localhost:6060/debug/pprof/profile?seconds=30'   # CPU
go tool pprof -http=:9999 http://localhost:6060/debug/pprof/heap                   # memory
curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=2' | head -100          # goroutine leak? deadlock?

(kubectl port-forward connects via the pod's network namespace, so it reaches 127.0.0.1:6060 inside the pod.)

Continuous profiling (Grafana Pyroscope, Parca, Datadog) samples all pods all the time at ~1–2% overhead. You can compare flame graphs from "before the deploy" and "after the deploy" without reproducing anything.

33.5 Incident workflow and postmortems

1. Declare an incident early, with roles (incident commander, comms, operator). It's cheap to downgrade.
2. Mitigate: roll back, scale up, fail over, shed load, disable a feature flag.
3. Communicate on a cadence: status page, stakeholders.
4. Resolve, then write a blameless postmortem within a week: timeline, impact (in SLO terms), contributing factors (plural: there is never one root cause), what went well, action items with owners. Action items should change the system, not "be more careful".

Reference: the public Kubernetes failure stories collection (k8s.af) is a catalogue of real postmortems: DNS, OOM, CPU limits, etcd, admission webhooks, ingress. Reading ten of them teaches you more about production Kubernetes than any certification.