32. SLOs and burn-rate alerting

32.1 Define the SLOs

SLI	Definition (good events / valid events)	SLO (30d)
Orders availability	non-5xx responses to `POST /orders` / all	99.9%
Orders latency	responses < 250 ms / all	99%
Order fulfilment	orders `paid` within 60 s of creation / orders created	99.5%

Error budget = 1 − SLO. At 99.9% over 30 days that's 43.2 minutes of full outage, or 0.1% of requests failing.

32.2 Multi-window, multi-burn-rate alerts (Google SRE Workbook)

Burn rate = how fast you're spending the budget relative to "exactly use it all in 30 days". A burn rate of 14.4 over 1 hour spends 2% of the monthly budget in that hour.

Severity	Long window	Short window	Burn rate	Budget consumed when it fires
page	1h	5m	14.4	2%
page	6h	30m	6	5%
ticket	1d	2h	3	10%
ticket	3d	6h	1	10%

The short window makes the alert reset quickly once the problem is fixed. The long window keeps it from firing on a 30-second blip.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: orbit-slo-orders, namespace: monitoring }
spec:
  groups:
    - name: slo.orders.availability.recording
      rules:
        - record: slo:orders_errors:ratio_rate5m
          expr: |
            sum(rate(http_server_request_duration_seconds_count{job="orbit/orders",http_response_status_code=~"5.."}[5m]))
            / sum(rate(http_server_request_duration_seconds_count{job="orbit/orders"}[5m]))
        # ... identical rules for 30m, 1h, 2h, 6h, 1d, 3d (generate them; see below)
    - name: slo.orders.availability.alerts
      rules:
        - alert: OrdersErrorBudgetBurnFast
          expr: |
            (slo:orders_errors:ratio_rate1h > (14.4 * 0.001) and slo:orders_errors:ratio_rate5m > (14.4 * 0.001))
            or
            (slo:orders_errors:ratio_rate6h > (6 * 0.001) and slo:orders_errors:ratio_rate30m > (6 * 0.001))
          labels: { severity: page, service: orders }
          annotations:
            summary: "orders is burning its 99.9% availability error budget fast"
            dashboard: https://grafana.acme.example/d/orbit-red?var-job=orbit/orders
            runbook_url: https://runbooks.acme.example/orbit/orders-availability
        - alert: OrdersErrorBudgetBurnSlow
          expr: |
            (slo:orders_errors:ratio_rate1d > (3 * 0.001) and slo:orders_errors:ratio_rate2h > (3 * 0.001))
            or
            (slo:orders_errors:ratio_rate3d > 0.001 and slo:orders_errors:ratio_rate6h > 0.001)
          labels: { severity: ticket, service: orders }
          annotations: { summary: "orders error budget burning steadily" }

Latency SLO uses the histogram bucket at the threshold (250 ms = the le="0.25" bucket boundary in the OTel default buckets):

1 - (
  sum(rate(http_server_request_duration_seconds_bucket{job="orbit/orders",le="0.25"}[1h]))
  / sum(rate(http_server_request_duration_seconds_count{job="orbit/orders"}[1h]))
)   # = fraction of SLOW requests: compare against (1 - 0.99) * burn rate

Best practice: don't hand-write these. Sloth or Pyrra take a 10-line SLO spec and generate all recording rules, alerts and a dashboard. OpenSLO is the vendor-neutral spec format.

# Sloth spec (generates ~30 rules)
version: prometheus/v1
service: orders
slos:
  - name: availability
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_server_request_duration_seconds_count{job="orbit/orders",http_response_status_code=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_server_request_duration_seconds_count{job="orbit/orders"}[{{.window}}]))
    alerting:
      page_alert: { labels: { severity: page } }
      ticket_alert: { labels: { severity: ticket } }

⚠️ Caveat (verify): the latency SLI assumes a histogram bucket boundary at exactly 0.25 s (the OTel default HTTP buckets). Confirm with count by (le) (http_server_request_duration_seconds_bucket). Check the Sloth spec against the Sloth version you run.

32.3 Error budget policy: the part that changes behavior

An SLO only matters if it drives decisions. Write it down and agree on it with product owners:

Budget remaining > 50%: ship freely; run chaos experiments in prod.
Budget < 25%: releases need extra review; prioritize reliability work.
Budget exhausted: feature freeze for the service except reliability fixes, until the 30-day window recovers.

Part IV checkpoint: run the k6 test, then break something (scale Postgres to 1 instance and kill it, or add a time.Sleep to orders). Go from the burn-rate alert → RED dashboard → an exemplar → the Tempo trace → that trace's logs in Loki, without typing a pod name.