Chapter 47 — Alerts that wake you for the right reasons

In one sentence

Alert on symptoms users feel, make every page actionable, and suppress the alerts that are just consequences of another alert.

The problem

The natural instinct is to alert on everything you can measure. Do that and you get alert fatigue: hundreds of notifications, all ignored, including the one that mattered.

The rules

1. Alert on symptoms, not causes.

   GOOD:  "p99 latency above 2 s for 5 minutes"       users are suffering
   GOOD:  "error rate above 1% for 3 minutes"
   GOOD:  "the site is unreachable from 2 of 5 regions"

   POOR:  "CPU above 80%"           So what? Is anything actually broken?
   POOR:  "interface at 75%"        Is this worth waking someone at 3am?
   POOR:  "disk at 70%"             That's a ticket, not a page.

High CPU with happy users is not an incident. It might be worth a dashboard or a ticket — not a page.

2. Every page must be actionable. If the person woken up can't do anything about it right now, it should not be a page. Make it a ticket or leave it on a dashboard.

3. Use duration, not instants. "Above threshold for 5 minutes" eliminates the vast majority of false alarms from momentary spikes.

4. Suppress dependent alerts. If the core switch is down, you do not want 400 separate alerts for everything behind it. Alert on the root cause and summarise the rest.

5. Distinguish "page" from "ticket".

   PAGE (wake a human now):   users are affected and it won't self-heal
   TICKET (look tomorrow):    a disk will fill in three weeks;
                              a redundant link failed but the backup is carrying it

The alerts everybody should have

Regardless of what you run:

   [ ] TLS CERTIFICATE EXPIRY -- warn at 30, 14, and 7 days.
       The most preventable outage in existence, and it still happens weekly
       to large companies.
   [ ] Domain name expiry
   [ ] The service is unreachable FROM OUTSIDE (probe from elsewhere)
   [ ] Error rate above your normal baseline
   [ ] p99 latency above your normal baseline
   [ ] DNS resolution failing
   [ ] Interface errors increasing (this precedes packet loss -- Chapter 35)
   [ ] Disk full on anything that writes logs
   [ ] Backups did not complete

A monitoring stack you could actually build

   METRICS      Prometheus + node_exporter (hosts) + snmp_exporter (devices)
   PROBES       blackbox_exporter (ICMP / TCP / HTTP / DNS / TLS expiry)
   DASHBOARDS   Grafana
   ALERTS       Alertmanager -> email / Slack / PagerDuty
   FLOWS        nfdump or ntopng, or your cloud's VPC flow logs
   LOGS         journald/syslog -> Loki, correlated by timestamp
   PACKETS      a tcpdump ring buffer on important hosts (Chapter 31)

All of the above is free and open source, and it runs comfortably on one small server for a modest environment.

Practice (25 min)

A. Write the certificate-expiry check — the single highest-value alert:

#!/bin/bash
for H in example.com yoursite.com; do
  END=$(echo | openssl s_client -connect $H:443 -servername $H 2>/dev/null \
        | openssl x509 -noout -enddate | cut -d= -f2)
  # macOS uses a different date syntax; try both
  EXP=$(date -d "$END" +%s 2>/dev/null || date -jf '%b %d %T %Y %Z' "$END" +%s)
  DAYS=$(( (EXP - $(date +%s)) / 86400 ))
  printf '%-25s %4d days\n' "$H" "$DAYS"
  [ $DAYS -lt 14 ] && echo "  *** WARNING: renew soon ***"
done

Put it in cron. You have now prevented a category of outage.

B. Establish your baselines. For a service you care about, record over a normal day: typical latency (p50/p95/p99), typical error rate, typical throughput. Without a baseline, no threshold you choose is meaningful.

C. Design alerts for the five problems in Chapter 35. For each — DNS failure, MTU black hole, firewall drop, packet loss, bufferbloat — write down: what symptom would you alert on, and what would the alert message say to help the responder?

Common confusions

"More monitoring is better." More signal is better. More alerts is usually worse.
"We should alert on every error." Alert on error rate relative to normal. Some errors are normal.
"Monitoring is for big companies." A cron job that checks your site from outside and emails you is monitoring, and it's better than nothing by a huge margin.

Check yourself

Why alert on symptoms rather than causes?
What's the difference between a page and a ticket?
Why require a threshold to be exceeded for a duration?
Name three alerts every service should have.

End of Part 11 — Milestone check

I can name the four kinds of monitoring signal
I know why application logs miss connection-level failures
I can explain why averages hide problems
I have read SNMP counters or kernel counters from a real device
I have written a layered probe script
I have a certificate-expiry check running