Week 15 Mini Project: Production-Hardened Gateway

Take the Week 12 Unified Security Proxy and add all production-hardening features.

Feature Addition Checklist

• Health check framework
  • /healthz — liveness (always fast, no upstream checks)
  • /readyz — readiness (checks all required upstream connections)
  • /startup — startup probe (wait for initialization to complete)
  • Per-subsystem checks: WAF rule load, TLS cert validity, upstream TCP reachability
• 3-stage graceful shutdown
  • Stage 1: Stop accepting (immediate)
  • Stage 2: Drain in-flight (30s budget)
  • Stage 3: Force-close + run hooks (5s budget)
• Signal handlers
  • SIGHUP → reload config from file (validate first, keep old if invalid)
  • SIGUSR1 → rotate log file
  • SIGUSR2 → dump goroutine stacks to /tmp/goroutine-dump-{timestamp}.txt
  • SIGTERM/SIGINT → graceful shutdown
• W3C trace context propagation
  • Inject traceparent on ingress if missing
  • Propagate to upstream via Director in reverse proxy
  • Record spans for each middleware layer
  • /debug/traces — last 1000 traces as JSON
• Chaos middleware (disabled by default)
  • Configurable error rate, delay rate, connection close rate
  • Enable via config file (not env var — must be hot-reloadable)
• Leak validation (automated)
  • Post-startup goroutine baseline
  • Expose goroutine_count in /debug/vars
  • Alert (log) if count grows > 2× baseline
• Zero-downtime restart
  • SIGUSR1 triggers FD inheritance restart
  • Old process drains; new process serves

Operational Test Commands

# Send SIGHUP to reload config
kill -HUP $(pgrep gateway)

# Send SIGUSR1 to rotate logs
kill -USR1 $(pgrep gateway)

# Send SIGUSR2 to dump goroutines
kill -USR2 $(pgrep gateway)
cat /tmp/goroutine-dump-*.txt | head -50

# Trigger zero-downtime restart
kill -USR1 $(pgrep gateway)  # starts new process
# Old process drains; new process inherits listener

# Health checks
curl http://localhost:8080/healthz
curl http://localhost:8080/readyz

Engineer Takeaway

This week implements what SRE teams call "Day 2 operations" — everything needed to run a service safely in production for months and years. SRE teams at Google, Stripe, and Cloudflare require all of these before a service enters production. A service without graceful shutdown, health checks, and signal handling is a liability, not an asset.