21. Health checks: probes done right

Probes are how Kubernetes decides to restart a container and whether to send it traffic. Get them wrong and Kubernetes' self-healing becomes self-harm.

21.1 The three probes

Mechanisms: httpGet (2xx/3xx = pass), tcpSocket, grpc (the standard gRPC health protocol), exec (command exit code; costly, forks a process every period).

21.2 Orbit's probes

startupProbe:
  httpGet: { path: /livez, port: http }
  periodSeconds: 2
  failureThreshold: 30          # up to 60s to start; liveness is held off until then
livenessProbe:
  httpGet: { path: /livez, port: http }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3           # 30s of consecutive failure before a restart
readinessProbe:
  httpGet: { path: /readyz, port: http }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2           # out of rotation within ~10s
  successThreshold: 1

Time to action = periodSeconds × failureThreshold (+ timeouts). Pick numbers by asking "how long can this be broken before acting is better than waiting?"

21.3 Rules (each one is a past outage)

1. Liveness must never check dependencies. If /livez pings the database, a 30-second DB failover restarts every pod of every service at once. They all reconnect to the recovering DB simultaneously (thundering herd), it falls over again, and you have a self-inflicted outage loop.
2. Liveness and readiness are different endpoints with different semantics. Liveness: "the event loop/HTTP server responds". Readiness: "I can do useful work".
3. Readiness and dependencies: be deliberate. If every replica's readiness depends on a shared dependency, a blip of that dependency removes all endpoints. Clients then get connection errors instead of a fast, meaningful 503. Common compromise: readiness checks only local conditions plus hard dependencies without which every request fails. Use timeouts, circuit breakers and fallbacks (serve stale cache) for the rest. Orbit's orders includes Postgres (no orders without it). catalog would not include Redis, since a cache miss is just slower.
4. Use a startupProbe for slow starters (JVM warm-up, cache preload, ML model load) instead of a huge initialDelaySeconds on liveness.
5. Probes must be cheap and fast. No DB queries that scan tables, no synchronous calls to other services. A probe that times out under load restarts healthy-but-busy pods exactly when you need them most.
6. Timeouts must tolerate GC pauses and CPU throttling. A 1 s timeout on a CPU-throttled pod (§23.3) is a restart generator.
7. Readiness goes false during shutdown (our SetDraining()), so endpoints are removed even if they haven't been yet.

War story: a payments company added a "deep health check" to liveness that verified Kafka, Postgres and two downstream APIs. During a partial Kafka incident, liveness failed fleet-wide and Kubernetes restarted ~400 pods in two minutes. Restart storms saturated the image registry and the database connection limits. The Kafka incident lasted 3 minutes; the self-inflicted outage lasted 40. Liveness went back to "return 200".

21.4 gRPC services

livenessProbe:
  grpc: { port: 9090, service: liveness }     # implement grpc.health.v1.Health
readinessProbe:
  grpc: { port: 9090, service: readiness }