31. Metrics, dashboards, and alerting

31.1 PromQL you will actually use

# --- RED for Orbit services (OTel http.server.request.duration, in seconds) ---
# Rate (RPS) per service
sum by (job) (rate(http_server_request_duration_seconds_count{job=~"orbit/.*"}[5m]))
# Error ratio (5xx / all)
sum by (job) (rate(http_server_request_duration_seconds_count{job=~"orbit/.*",http_response_status_code=~"5.."}[5m]))
  / sum by (job) (rate(http_server_request_duration_seconds_count{job=~"orbit/.*"}[5m]))
# p95 / p99 latency (ALWAYS aggregate buckets by le before histogram_quantile)
histogram_quantile(0.95, sum by (job, le) (rate(http_server_request_duration_seconds_bucket{job=~"orbit/.*"}[5m])))

# --- Business ---
sum(rate(orbit_orders_created_total[5m])) * 60                         # orders/minute
histogram_quantile(0.95, sum by (le) (rate(orbit_worker_process_duration_seconds_bucket[5m])))

# --- Kubernetes health (kube-state-metrics + cAdvisor) ---
sum by (namespace, pod) (increase(kube_pod_container_status_restarts_total{namespace="orbit"}[1h])) > 0
kube_pod_container_status_last_terminated_reason{reason="OOMKilled", namespace="orbit"}
sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0
kube_deployment_status_replicas_available / kube_deployment_spec_replicas < 1
kube_horizontalpodautoscaler_status_current_replicas >= kube_horizontalpodautoscaler_spec_max_replicas   # HPA maxed out
# memory usage vs limit, per container
max by (pod, container) (container_memory_working_set_bytes{namespace="orbit",container!=""})
  / max by (pod, container) (kube_pod_container_resource_limits{namespace="orbit",resource="memory"})
# PVC will be full within 4 days?
predict_linear(kubelet_volume_stats_available_bytes{namespace="orbit"}[6h], 4 * 24 * 3600) < 0

# --- USE for nodes (node-exporter) ---
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))      # CPU utilization
node_pressure_cpu_waiting_seconds_total                                    # PSI saturation (rate it)

PromQL rules of thumb:

rate() on counters, then sum. Never sum and then rate.
Range ≥ 4× the scrape/export interval ([1m] for 15 s, [5m] is a safe default).
histogram_quantile needs le in the by clause, and you can't average percentiles across pods. Aggregate buckets instead.

31.2 Recording rules: precompute what dashboards and alerts reuse

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: orbit-recording, namespace: monitoring }
spec:
  groups:
    - name: orbit.red
      interval: 30s
      rules:
        - record: job:http_requests:rate5m
          expr: sum by (job) (rate(http_server_request_duration_seconds_count{job=~"orbit/.*"}[5m]))
        - record: job:http_errors:ratio_rate5m
          expr: |
            sum by (job) (rate(http_server_request_duration_seconds_count{job=~"orbit/.*",http_response_status_code=~"5.."}[5m]))
            / sum by (job) (rate(http_server_request_duration_seconds_count{job=~"orbit/.*"}[5m]))
        - record: job:http_latency:p95_5m
          expr: histogram_quantile(0.95, sum by (job, le) (rate(http_server_request_duration_seconds_bucket{job=~"orbit/.*"}[5m])))

31.3 Alert rules: page on symptoms, ticket on causes

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: orbit-alerts, namespace: monitoring }
spec:
  groups:
    - name: orbit.alerts
      rules:
        - alert: OrbitWorkerBacklogGrowing
          expr: |
            sum(rate(orbit_orders_created_total[10m])) > 0
            and sum(rate(orbit_worker_processed_total{outcome="paid"}[10m])) < 0.8 * sum(rate(orbit_orders_created_total[10m]))
          for: 15m
          labels: { severity: ticket, service: worker }
          annotations:
            summary: "Worker processes < 80% of incoming orders for 15m"
            runbook_url: https://runbooks.acme.example/orbit/worker-backlog
        - alert: OrbitPodCrashLooping
          expr: increase(kube_pod_container_status_restarts_total{namespace=~"orbit.*"}[15m]) > 3
          for: 5m
          labels: { severity: ticket }
          annotations:
            summary: "{{ $labels.pod }} restarted {{ $value | humanize }} times in 15m"
            runbook_url: https://runbooks.acme.example/k8s/crashloop
        - alert: OrbitOOMKilled
          expr: increase(container_oom_events_total{namespace=~"orbit.*"}[30m]) > 0
          labels: { severity: ticket }
          annotations: { summary: "{{ $labels.pod }}/{{ $labels.container }} OOMKilled" }
        - alert: OrbitHPAMaxedOut
          expr: kube_horizontalpodautoscaler_status_current_replicas{namespace=~"orbit.*"} >= kube_horizontalpodautoscaler_spec_max_replicas{namespace=~"orbit.*"}
          for: 30m
          labels: { severity: ticket }
          annotations: { summary: "{{ $labels.horizontalpodautoscaler }} at maxReplicas for 30m: raise the max or find the bottleneck" }
        - alert: OrbitPostgresConnectionsHigh
          expr: sum(cnpg_backends_total{namespace=~"orbit.*"}) / max(cnpg_pg_settings_setting{name="max_connections",namespace=~"orbit.*"}) > 0.8
          for: 10m
          labels: { severity: ticket, service: postgres }
          annotations: { summary: "Postgres connections > 80% of max_connections" }

What makes a good alert:

Page only on user-visible symptoms (SLO burn, §32). "CPU at 90%" is not a page. If users are fine, it can wait until morning.
Every alert has a runbook URL with what it means, how to verify, and how to mitigate.
for: durations suppress flapping on transient blips.
Every page must be actionable. If the on-call's response is "ack and ignore", delete the alert or downgrade it to a ticket. Alert fatigue is how real pages get missed.

⚠️ Caveat (verify): every query here assumes the metric names from the §29 caveat and the label promotion from the §30.1 caveat. Exporter metric names (cnpg_backends_total, cnpg_pg_settings_setting, container_oom_events_total) also vary by exporter version. Check each in Prometheus's Explore view and run promtool check rules on the extracted rule groups before deploying.

31.4 The dead man's switch

kube-prometheus-stack ships an always-firing Watchdog alert. We route it to an external heartbeat service (healthchecks.io, Cronitor, PagerDuty's heartbeat). If the heartbeats stop, the external service pages you. That's the only way to learn your Prometheus/Alertmanager pipeline itself is dead.

31.5 Dashboards as code

apiVersion: v1
kind: ConfigMap
metadata:
  name: orbit-red-dashboard
  namespace: monitoring
  labels: { grafana_dashboard: "1" }   # the Grafana sidecar loads it automatically
data:
  orbit-red.json: |
    { "title": "Orbit / RED", "uid": "orbit-red", "schemaVersion": 39,
      "templating": { "list": [ { "name": "job", "type": "query", "datasource": "prometheus",
          "query": "label_values(http_server_request_duration_seconds_count{job=~\"orbit/.*\"}, job)" } ] },
      "panels": [
        { "type": "timeseries", "title": "Requests/s", "gridPos": {"x":0,"y":0,"w":8,"h":8},
          "targets": [ { "expr": "sum by (job) (rate(http_server_request_duration_seconds_count{job=~\"$job\"}[5m]))" } ] },
        { "type": "timeseries", "title": "Error ratio", "gridPos": {"x":8,"y":0,"w":8,"h":8},
          "targets": [ { "expr": "job:http_errors:ratio_rate5m{job=~\"$job\"}" } ] },
        { "type": "timeseries", "title": "p95 latency", "gridPos": {"x":16,"y":0,"w":8,"h":8},
          "targets": [ { "expr": "histogram_quantile(0.95, sum by (job, le) (rate(http_server_request_duration_seconds_bucket{job=~\"$job\"}[5m])))", "exemplar": true } ] }
      ] }

Dashboard design that works during incidents:

Top row = the SLO/RED summary. Is it broken, and since when?
Next rows follow the request path: gateway → service → dependencies (DB, Redis, queue).
Then saturation: CPU throttling, memory vs limit, pool usage, HPA replicas vs max.
Add deploy annotations (Argo CD sync events) so "it started at the deploy" is visible at a glance.

Import community dashboards as a starting point: Node Exporter Full (1860), the CloudNativePG dashboard, Envoy Gateway's, and Go runtime metrics.