Part 17 -- Kubernetes in practice

You know the pieces now: Pods, Deployments, Services, ConfigMaps, scheduling. This part is about the muscle memory of actually operating a cluster -- reading its state when something is wrong, rolling out changes safely, and packaging applications so other people can install them without reading your mind.

Chapter 59 -- Reading cluster state and debugging a pod that won't start

In one sentence

kubectl describe, kubectl logs, and kubectl get events are the stethoscope, thermometer, and chart review of Kubernetes debugging, and almost every "pod won't start" problem falls into one of about six buckets.

Why this matters

A Deployment says "3 desired, 1 available" and nothing works, and the instinct of someone new to Kubernetes is to restart everything or delete the cluster and start over. That instinct is expensive and usually wrong. Kubernetes tells you exactly what is wrong, in plain English, in the Events section of kubectl describe pod -- you just have to know to look there first, before logs, before dashboards, before Slack.

For a manager: the cost of a production incident is often dominated not by the failure itself but by the time-to-diagnosis. An engineer who knows the six buckets below finds the cause in two minutes. An engineer who does not know them tries five unrelated things over 40 minutes. That gap, multiplied across every incident a team has in a year, is a real and measurable amount of engineering time -- which is why "can you debug a broken pod calmly" is a legitimate, common interview question for SRE and platform roles.

How it actually works

The debugging order that works, almost every time:

   1. kubectl get pods                 -- what STATUS is it in?
   2. kubectl describe pod <name>      -- read the Events at the bottom
   3. kubectl logs <name>              -- what did the app itself say?
   4. kubectl logs <name> --previous   -- what did it say before it
                                          last crashed? (for CrashLoopBackOff)
   5. kubectl get events --sort-by=.metadata.creationTimestamp
                                       -- cluster-wide event timeline

The six buckets, by STATUS column in kubectl get pods:

   Pending
     The Pod has not been scheduled to a node yet. Cause is almost
     always in `describe`'s Events: "Insufficient cpu", "Insufficient
     memory", "0/5 nodes are available: 5 node(s) had taint
     {node-role: master} that the pod didn't tolerate", or a
     PersistentVolumeClaim that has no matching PersistentVolume.
     Fix: right-size requests, add tolerations, or fix storage.

   ImagePullBackOff / ErrImagePull
     Kubernetes cannot pull the container image. Almost always a typo
     in the image name/tag, a private registry with no imagePullSecret
     configured, or (surprisingly common) the image tag was deleted or
     overwritten by a later build.
     Fix: `kubectl describe pod` shows the exact registry error.

   CrashLoopBackOff
     The container starts, then exits, repeatedly, with increasing
     backoff delay between restarts. The container image and pull are
     fine -- the *application* is crashing. This is the one where
     `kubectl logs --previous` is essential, because by the time you
     look, the current attempt's logs may be empty (crashed before
     printing anything) while the previous attempt's logs show the
     real stack trace.
     Common causes: missing environment variable / Secret, cannot
     reach a database on startup, misconfigured config file, the
     entrypoint script has a typo.

   Pending forever with "0/N nodes are available: N Insufficient memory"
     This is a scheduling problem, not a crash. Either lower the
     Pod's memory *request*, add nodes, or find and evict a
     resource-hogging Pod that's wasting reserved-but-unused capacity.

   OOMKilled (seen in `kubectl describe pod`, "Last State: Terminated,
   Reason: OOMKilled")
     The container exceeded its memory *limit* and the kernel's OOM
     killer (Chapter 11) killed it. Fix: raise the limit if the
     workload legitimately needs more memory, or fix a memory leak if
     it doesn't.

   Running, but Service traffic still fails (0/1 Ready)
     The container process is alive but the readiness probe is
     failing -- e.g. the app takes 45 seconds to warm up but the
     probe's initialDelaySeconds is 5. Kubernetes is correctly
     refusing to send traffic to a Pod that says it isn't ready;
     the fix is almost always in the probe configuration.

Real-world example

A well-known and often-retold Kubernetes debugging story from the SRE community involves a service that worked perfectly in staging and then went into CrashLoopBackOff immediately after every production deploy. kubectl logs on the current, crashed container showed nothing -- an empty log, because the process died within milliseconds of starting, before its logging library had even finished initializing. Only kubectl logs --previous, combined with describe pod's "Last State" block, revealed the actual error: the production ConfigMap was missing a key that staging's ConfigMap happened to have, so the app's config parser threw an unhandled exception at import time, before any log line could be written. The lesson generalized into a common piece of team tooling: runbooks for "Pod won't start" now routinely open with the exact five-step sequence above, in order, precisely because skipping straight to "check the logs" misses the crash that happens before there are any logs to check. Datadog, Google Cloud's SRE documentation, and the official Kubernetes troubleshooting docs all converge on the same ordered checklist independently, which is itself a signal that this is the sequence that actually works in practice rather than a stylistic preference.

Try it yourself

# Deliberately break a pod, then debug it using only the sequence above.
kubectl create deployment broken --image=nginx:this-tag-does-not-exist
kubectl get pods
# STATUS should show ImagePullBackOff or ErrImagePull within ~30s

kubectl describe pod -l app=broken | tail -20
# read the Events block -- it names the exact problem

kubectl delete deployment broken

# Now break it a different way: a container that crashes on startup.
kubectl run crasher --image=busybox --restart=Never -- sh -c "exit 1"
kubectl get pod crasher
kubectl logs crasher            # probably empty or minimal
kubectl describe pod crasher | grep -A5 "Last State"
kubectl delete pod crasher

Common mistakes

Jumping to kubectl logs before kubectl describe, and missing a scheduling or image-pull problem that never produced any application logs at all because the container never ran.
Reading only the current logs of a crash-looping container and concluding "no errors," when --previous would show the actual crash.
Treating Pending as "the cluster is broken" when it usually means "this specific Pod's resource requests do not fit anywhere right now" -- a request/capacity problem, not an outage.
Restarting the Deployment (kubectl rollout restart) as a first response before reading a single event -- this sometimes "fixes" a transient problem by luck, teaches the team nothing, and can mask a real bug that will recur.

Check yourself

A Pod shows STATUS: Pending and describe says "Insufficient cpu." What are two different valid fixes, and what's the trade-off between them?
Why can kubectl logs show nothing at all for a container that is definitely crashing, and what command reveals the real error instead?
A Pod is Running and 1/1 Ready, but a Service pointing at it times out. Name two places, other than the Pod's own health, that could be the actual cause.