Frontend Labs
⋯

46. Pitfalls that cause real outages

Pitfall Symptom Fix (section)
Shell-form CMD / no SIGTERM handling slow stops, dropped requests on deploy exec form, signal handling (§3.3)
No preStop / instant shutdown 502s on every rollout preStop sleep + drain (§13.5)
Liveness checks dependencies fleet-wide restart storm during a DB blip liveness = process health only (§21.3)
CPU limits on latency-sensitive apps p99 spikes at low average CPU requests only, or generous limits (§23.3)
Runtime not container-aware OOMKilled loops GOMEMLIMIT / JVM / Node flags (§23.4)
No requests set noisy neighbors, BestEffort evictions, broken HPA measured requests (§23.6)
All replicas on one node/zone outage on a single node reboot topology spread (§24.1)
PDB that allows 0 evictions stuck cluster upgrades allow ≥ 1 (§22.3)
:latest / mutable tags can't tell what's running; failed rollbacks sha tags + digests (§10.1)
ndots:5 + external calls DNS latency, CoreDNS overload FQDN dots, ndots 2, NodeLocal DNS (§15.3)
Long-lived gRPC/HTTP2 connections new pods idle after scale-out client LB / max connection age (§15.4)
HPA + replicas in Git GitOps resets replicas every sync remove replicas (§27.2)
Pods × pool size > DB max_connections "too many connections" at peak scale pooler / pool math (§27.4)
Unbounded metric labels Prometheus OOM, monitoring outage bounded attributes (§29.4)
Untested backups data loss when you need them automated restore tests (§39.4)
Locking migration at peak full outage behind one ALTER expand/contract, lock_timeout (§38)
Admission webhook down + Fail nothing can be created exclusions, HA, timeouts (§34.5)
Prune without protection stateful resource deleted by GitOps Prune=false on stateful (§36.5)
Pods can reach instance metadata cloud credential theft via SSRF egress policy, IMDSv2, workload identity (§34.3)

Generated from real-life-k8s-guide.md · 8 Oct 2026 16:36