Chapter 60 -- Deployment patterns: rolling updates, health checks, rollbacks

In one sentence

A Deployment's default rolling update replaces old Pods with new ones a few at a time, gated entirely by readiness probes -- which means a deployment is only as safe as the health check backing it, and a bad health check turns "safe by design" into "outage by design."

Why this matters

This is the single most consequential piece of Kubernetes configuration most teams under-invest in. The rollout mechanics (maxSurge, maxUnavailable, readiness gating) are genuinely well engineered and, when paired with a correct readiness probe, make bad deploys self-limiting: Kubernetes stops rolling forward the moment new Pods fail to become ready, and traffic never reaches them. But if the readiness probe just checks "is the process running" instead of "can this process actually serve a request," the safety net has a hole in it exactly where it's needed most -- and the org discovers this precisely during the deploy that matters.

How it actually works

   ROLLING UPDATE, Deployment with replicas: 4,
   maxSurge: 1, maxUnavailable: 1 (the defaults, roughly):

   Start:   [v1] [v1] [v1] [v1]                4 old, 0 new
   Step 1:  [v1] [v1] [v1] [v1] [v2]            surge: create 1 new
                                    ^ wait for readiness probe to pass
   Step 2:  [v1] [v1] [v1]      [v2] [v2]      1 old terminated, 1 more new
   Step 3:  [v1] [v1]          [v2] [v2] [v2]
   Step 4:  [v1]               [v2] [v2] [v2] [v2]
   Step 5:                     [v2] [v2] [v2] [v2]   done

   At every step, the Service only ever sends traffic to Pods that
   are Ready. A new Pod that fails its readiness probe simply never
   receives traffic and the rollout STALLS there -- it does not
   proceed to kill more old Pods. This is the core safety property.

Two different probes, often confused:

   livenessProbe:   "is this process healthy, or should it be
                      killed and restarted?" A failing liveness
                      probe causes a container restart. Wrong
                      liveness config -> death spiral: probe fails
                      under normal load -> container restarted ->
                      cold-starts and is slow -> fails probe again.

   readinessProbe:  "should this Pod currently receive traffic?"
                      A failing readiness probe removes the Pod from
                      the Service's endpoint list WITHOUT killing it.
                      This is the one that gates rollouts and is safe
                      to make strict, because failing it doesn't
                      destroy anything -- it just stops sending new
                      requests until the Pod says it's ready again.

Rollback is a first-class operation, not a manual redeploy:

kubectl rollout history deployment/api
kubectl rollout undo deployment/api              # back to previous revision
kubectl rollout undo deployment/api --to-revision=3
kubectl rollout status deployment/api            # watch a rollout live

Real-world example

A widely cited pattern in postmortems (Monzo, Cloudflare, and many smaller companies' public engineering blogs describe variants of this) is the "healthy-looking crash": a service's /healthz endpoint returns 200 as long as the HTTP server thread is alive, but the service depends on a database connection pool that can silently exhaust. The liveness probe keeps passing because the HTTP server itself never dies -- it just returns 500s to every real request. Kubernetes has no idea anything is wrong: the container isn't restarted (liveness passes) and the rollout isn't blocked (readiness, if it exists at all, also just checks the HTTP server). The fix pattern that shows up repeatedly across these postmortems is the same: separate liveness (cheap, "is the process alive") from readiness (meaningful, "can I actually reach my database and process a real request"), and make the readiness probe do at least one lightweight dependency check. Kubernetes's own documentation explicitly warns against conflating the two probes for exactly this reason -- it is one of the most common production misconfigurations in the ecosystem, common enough that it appears in nearly every "Kubernetes gotchas" talk given at KubeCon.

Try it yourself

kubectl create deployment demo --image=nginx --replicas=4
kubectl set image deployment/demo nginx=nginx:1.25
kubectl rollout status deployment/demo
kubectl rollout history deployment/demo

# simulate a bad deploy and watch Kubernetes refuse to finish rolling out
kubectl set image deployment/demo nginx=nginx:this-tag-does-not-exist
kubectl rollout status deployment/demo --timeout=20s
# it will report the rollout is stuck -- old Pods are still serving

kubectl rollout undo deployment/demo
kubectl rollout status deployment/demo
kubectl delete deployment demo

Build it in Go

A Go service needs three things to make rolling updates seamless: a /readyz that fails as soon as SIGTERM arrives, a delay before srv.Shutdown, and Shutdown itself to drain in-flight requests. The TCP/IP guide's lbdrain lab proves the difference under load, and Chapter 82 here has the matching Kubernetes manifest (probes, preStop, terminationGracePeriodSeconds).

Common mistakes

Using the same probe definition (or no readiness probe at all) for both liveness and readiness, so a dependency outage looks "healthy" to Kubernetes.
Setting initialDelaySeconds too low for a slow-starting app (JVM services are the classic offender), causing the liveness probe to kill the container mid-startup, forever, before it ever gets a chance to finish booting.
Setting maxUnavailable: 0 and maxSurge: 0 (accidentally, via a misunderstood template) which makes rollouts impossible to progress at all, and the Deployment just hangs.
Assuming kubectl rollout undo also rolls back a database migration that shipped in the same release -- it only reverts the Deployment's Pod template, not any side effect the new code already caused.

Check yourself

What is the actual, mechanical difference between a livenessProbe failure and a readinessProbe failure -- what does Kubernetes do in each case?
A service's /healthz always returns 200 even when its database is unreachable. What kind of incident does this specifically enable, and why doesn't Kubernetes catch it?
Why does a rolling update with maxUnavailable: 1 on a Deployment with 4 replicas stall, rather than fail outright, when new Pods can't become ready?