53. Walkthrough: Design a Metrics Monitoring & Alerting Platform

Step 0 — Clarify the business goal

🎤 Candidate: Is this an internal platform for our engineering org, or a SaaS product? Internal means the outcome is lower incident detection and resolution time (MTTD/MTTR) and cost. SaaS adds tenant isolation and billing.

🧑‍💼 Interviewer: Internal, for ~2,000 engineers. Today's system falls over during big incidents, exactly when it's needed.

🎤 Candidate: Then the key requirement is that alerting keeps working during the incidents it's meant to detect: under ingestion spikes, during regional failures, and when people run expensive dashboards. I'll separate the alerting path from the ad-hoc query path.

Step 1–2 — Requirements

Step 3 — Estimates

1M sources × ~1,000 series each = 1B active series; sample every 10 s → 100M samples/s
Raw sample = 16 B (8 B ts + 8 B value) → 1.6 GB/s raw
Gorilla-style compression ≈ 1.4 B/sample → ~140 MB/s → ~12 TB/day compressed (before replication)
Series index: 1B series × ~200 B labels ≈ 200 GB → sharded in-memory inverted index
Head (recent 2 h) in memory: 100M samples/s × 7,200 s × 1.4 B ≈ 1 TB (× replication) → ~hundreds of nodes
Alert rules: 200k rules evaluated every 30 s → ~7k rule evaluations/s

Step 4 — Data model & APIs

Series identity: metric_name + sorted label set → series_id = hash(...)
  http_requests_total{service="checkout", region="eu1", code="500"}
Sample: (series_id, timestamp_ms, float64)
Inverted index: label=value → postings list of series_ids  (service="checkout" ∩ code="500")
Write:  POST /api/v1/push (batched, compressed protobuf; tenant header)
Query:  GET /api/v1/query_range?q=sum(rate(http_requests_total{code="500"}[5m])) by (service)&start&end&step
Rules:  YAML in Git → rule groups → evaluator

Step 5 — High-level architecture

 Agents (on every node): scrape/collect, pre-aggregate, drop banned labels, buffer to disk on failure
    │
    ▼
 Ingestion gateways (per region): auth, per-team rate & CARDINALITY limits, validation
    │ shard by hash(series_id) — replicate ×3
    ▼
 Ingesters: in-memory head block (last ~2 h) + WAL ──(every 2 h)──▶ immutable blocks → object storage
                                                                    │
                                     Compactor (merge, dedupe replicas, downsample 1m/1h, retention)
                                                                    │
 Query frontend (split by time, cache results, per-query limits) ─▶ Queriers ─▶ ingesters (recent)
                                                                            └▶ store gateways (historical blocks)
 ── ALERTING PATH (isolated) ──────────────────────────────────────────────────────────────
 Rule evaluators (sharded by rule group, HA pairs) ─▶ query ingesters DIRECTLY (recent data only)
    ─▶ Alertmanager cluster (dedupe HA duplicates, group, inhibit, silence, route) ─▶ pager/chat
 ── META-MONITORING ─────────────────────────────────────────────────────────────────────────
 A small, independent stack (different region/infra) watches the monitoring system itself

Step 6 — Deep dives

1. Cardinality is the main scaling and cost risk. One label with user IDs creates millions of series. Defenses: per-team active-series limits at the gateway (reject with a clear error), label allowlists for hot metrics, agent-side aggregation, and a "top series by cardinality" report for every team. Cost attribution per team changes behavior faster than policies do.
2. Write path durability: the WAL in ingesters plus replication factor 3 (quorum writes) means losing one ingester loses nothing. Some designs put Kafka in front as a durable buffer, which helps absorb spikes but adds latency and cost. Choose based on the spike profile.
3. The query path must not hurt alerting: dashboards scanning 30 days × 100k series can saturate queriers. Use separate query pools for alerting and humans, per-query limits (series touched, samples scanned, time range), downsampled data for long ranges, and result caching aligned to step boundaries.
4. Percentiles across hosts: store histograms (bucketed, or sparse/native histograms, or DDSketch), not per-host p99s, because percentiles can't be averaged (§16.5).
5. Alert quality: alert on SLO burn rates (§16.3), use for: durations and hysteresis to stop flapping, alert on absent data, and route by ownership metadata on services.

Step 7 — Failure modes & evolution