16. Observability, SLIs/SLOs & Error Budgets

16.1 The pillars

Metrics: cheap aggregates. Use RED (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for resources. See the scale guide §11.
Logs: structured (JSON), with a trace/request ID, sampled at high volume.
Traces: distributed request flows (OpenTelemetry). The only practical way to debug latency in a call graph.
Profiles: continuous CPU and heap profiling to find cost and latency hotspots.

16.2 SLIs, SLOs, SLAs

SLI  (indicator): what you measure   → "proportion of checkout requests served < 300 ms with 2xx"
SLO  (objective): internal target    → "99.9% of them over a rolling 28 days"
SLA  (agreement): contract + penalty → "99.5% monthly, or 10% service credit"
SLA < SLO, so you can miss the SLO without paying penalties.

Good SLIs are measured as close to the user as possible (load balancer or client telemetry, not the server's own view), and cover critical user journeys ("can pay", "can send a message"), not individual servers.

16.3 Error budgets

SLO 99.9% over 28 days → error budget = 0.1% of requests
At 10M requests/day → 280M requests per 28 days → 280,000 failed requests allowed

Budget healthy  → ship faster, run experiments, take risks
Budget exhausted → freeze risky launches, prioritize reliability work

The error budget turns "reliability vs velocity" from an argument into a policy. EM candidates should be ready to explain how they'd run it with product partners.

Burn-rate alerting (from the Google SRE Workbook): page when you're spending the budget too fast, not on every error spike.

Alert	Burn rate	Long window	Short window	Action
Fast burn	14.4× (2% of budget in 1 h)	1 h	5 min	Page
Medium burn	6× (5% in 6 h)	6 h	30 min	Page
Slow burn	1× (10% in 3 days)	3 days	6 h	Ticket

16.4 🏗️ Architecture: a telemetry pipeline

 Services (OpenTelemetry SDK: traces, metrics, logs; trace_id in every log line)
     │ OTLP
     ▼
 Node/sidecar collectors ── batch, add resource attrs (region, cell, version)
     │
     ▼
 Gateway collectors ── TAIL-BASED SAMPLING: keep 100% of error/slow traces, 1% of normal
     │                 ── metric cardinality limits, PII scrubbing
     ├──▶ Metrics TSDB (Prometheus/M3/Monarch) ──▶ SLO burn-rate rules ──▶ paging
     ├──▶ Trace store (Jaeger/Tempo)            ◀── exemplars link metric spikes to traces
     └──▶ Log store (hot 7d) ──▶ object storage (cold, 1y) for audits/investigations
 Separately: synthetic probes from outside the network + real-user monitoring (RUM)

Burn-rate alert evaluation:

 error_ratio(1h) > 14.4 × (1 − SLO)  AND  error_ratio(5m) > 14.4 × (1 − SLO)  → PAGE
 (long window = significance, short window = "still happening" → fast reset after recovery)

16.5 Deeper: details that separate good observability from expensive noise

Histograms, not averages, and not pre-computed percentiles. You can't average p99s across hosts. Store histogram buckets (or sketches such as DDSketch or HdrHistogram) and compute percentiles after aggregation. The scale guide §3.14 has the math.
Head vs tail sampling: head sampling decides at the start (cheap, but it misses rare errors). Tail sampling decides after the trace completes (keeps every error, but needs buffering).
Exemplars: attach a trace ID to a histogram bucket, so you can jump from "p99 spiked at 14:02" to one actual slow trace.
Symptoms, not causes: page on user-facing SLO burn. CPU at 90% isn't worth waking someone up if users are fine.

16.6 🌍 In production

Google Dapper (2010 paper) introduced large-scale distributed tracing with low-overhead sampling, and inspired Zipkin, Jaeger, and OpenTelemetry.
Google Monarch (VLDB 2020): a planet-scale in-memory time-series system, regionalized so monitoring keeps working when parts of the network don't.
Meta Scuba (2013): in-memory, real-time ad-hoc analysis of logs and events. Engineers slice by any dimension within seconds of an event.
Netflix SPS: the "stream starts per second" metric is compared against its own historical pattern. A deviation is the primary signal that something is wrong, whatever the cause. It's a business-level SLI.
Uber built Jaeger (tracing) and M3 (metrics) because off-the-shelf tools couldn't handle its cardinality and volume.

16.7 ⚠️ Corner cases & deep details

The monitoring depends on the monitored system: during the 2017 S3 outage, AWS's own status dashboard couldn't update for a while because it depended on S3. Host status pages and alerting outside the failure domain.
Missing data looks healthy: a service that stops emitting metrics shows zero errors. Alert on absence of data too.
Bot traffic in the SLI denominator: a scraper flood of cheap 200s improves your success ratio while real users suffer. Segment SLIs by traffic class.
Clock skew in traces: child spans appear to start before their parent. Tracing UIs need to tolerate skew, and you should compute latency from one clock per span.
Logging under incidents: error logs explode 100×, the log pipeline saturates, and backpressure blocks the application (synchronous logging). Use async, bounded, droppable log buffers, and rate-limit identical errors.
Alert fatigue: if more than ~2 pages per on-call shift are non-actionable, people start ignoring pages. Track the actionable rate as a team health metric (an EM topic).