22. Self-healing and auto-recovery

22.1 The layers of recovery

22.2 Faster node-failure recovery for stateless pods

By default every pod tolerates node.kubernetes.io/unreachable and not-ready for 300 s, so it waits 5 minutes before being replaced. For stateless services with ≥ 2 replicas that's far too long:

tolerations:
  - { key: node.kubernetes.io/unreachable, operator: Exists, effect: NoExecute, tolerationSeconds: 30 }
  - { key: node.kubernetes.io/not-ready,   operator: Exists, effect: NoExecute, tolerationSeconds: 30 }

Don't do this for StatefulSets with RWO volumes. A "dead" node might just be partitioned and still writing to the disk, and that's how you get split brain. Let the operator decide.

22.3 PodDisruptionBudgets

Voluntary disruptions (node drains for upgrades, Karpenter consolidation, cluster autoscaler scale-down) go through the Eviction API, which honors PDBs:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: orders }
spec:
  maxUnavailable: 1                                    # or minAvailable: 2 / "80%"
  selector: { matchLabels: { app.kubernetes.io/name: orders } }
  unhealthyPodEvictionPolicy: AlwaysAllow              # don't let already-broken pods block node drains

War story: minAvailable: 100% (or maxUnavailable: 0) on a 1-replica Deployment means a node can never be drained. A cluster upgrade hangs for hours until someone deletes the PDB by hand. PDBs must allow at least one eviction, and single-replica workloads need either 2 replicas or no PDB.

22.4 Retries, timeouts, and circuit breakers in the app

Kubernetes restarts things; it doesn't make individual requests succeed. Every outbound call needs:

// A reusable outbound HTTP client: always set timeouts; the zero-value http.Client has NONE.
var httpClient = &http.Client{
	Timeout: 3 * time.Second,
	Transport: &http.Transport{
		MaxIdleConnsPerHost: 100,          // default 2 causes connection churn under load
		IdleConnTimeout:     90 * time.Second,
	},
}

// Retry only idempotent operations, with capped exponential backoff + full jitter,
// and stop when the caller's context deadline is exceeded.
func retry(ctx context.Context, attempts int, fn func(context.Context) error) error {
	var err error
	for i := 0; i < attempts; i++ {
		if err = fn(ctx); err == nil || !isRetryable(err) {
			return err
		}
		backoff := time.Duration(rand.Int64N(int64(100*time.Millisecond) << i)) // full jitter
		select {
		case <-time.After(min(backoff, 2*time.Second)):
		case <-ctx.Done():
			return errors.Join(err, ctx.Err())
		}
	}
	return err
}

• Timeouts propagate: each hop's timeout must be smaller than its caller's. Use context.Context deadlines.
• Retry budgets: retries at every layer multiply. With 3 layers × 3 retries, one user request can become 27 at the bottom. That is a retry storm. Retry at one layer, or cap retries as a percentage of traffic.
• Circuit breakers (e.g. sony/gobreaker) fail fast when a dependency is down, so you don't hold goroutines and connections waiting for it.
• Load shedding: return 503 quickly under overload instead of queueing forever. A fast "no" beats a slow timeout.

22.5 Priority and preemption

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: orbit-critical }
value: 100000
description: "User-facing request path (catalog, orders)"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: orbit-batch }
value: 1000
preemptionPolicy: Never          # waits for room instead of evicting others

When the cluster is full, higher-priority pods preempt lower-priority ones. Put user-facing APIs above batch and workers, so a capacity crunch degrades background work first.