69. Active probing, synthetic monitoring, and alerting that works

69.1 Passive vs active

PASSIVE  (Ch 66-68)  observes real traffic
   + reflects real users      - only sees paths users actually take
                              - silent when there is no traffic (3am outage
                                goes unnoticed until morning)

ACTIVE   (this chapter)  generates synthetic traffic
   + constant baseline        - synthetic; may not match real user experience
   + tests paths before       - adds (small) load
     users do
   + detects "up but broken"

You need both. Active monitoring is what tells you a service is down at 3am; passive tells you what real users are experiencing.

69.2 Layered synthetic checks

Design probes to isolate the layer that failed — mirroring Ch. 34's method:

L3   ping / ICMP           reachability + RTT + loss
L3   traceroute/mtr        path and where loss begins
L4   TCP connect to :443   is the port open? handshake time?
L6   TLS handshake         certificate valid? expiry? handshake time?
L7   HTTP GET /healthz     status code, latency, body content check
L7   full transaction      log in, search, add to cart, check out
DNS  resolve the name      from multiple resolvers and regions

A check at each layer means the alert names the layer. A single "site is down" check tells you nothing about where.

69.3 Standardised network probing

One-way delay matters because paths are asymmetric (Ch. 13). A 90 ms round trip could be 20 ms out and 70 ms back — TWAMP shows that; ping cannot.

69.4 Where to probe from

inside the datacentre     -> tests the service, not the network to users
from each POP/region      -> catches regional routing/peering problems
from real user networks   -> external synthetic providers, RIPE Atlas,
                             or RUM (Real User Monitoring) in the page

RUM (real-user timing beaconed from browsers, via the Navigation Timing and Resource Timing APIs) is the ground truth for user experience; synthetics are the controlled, comparable baseline. Use synthetics for alerting and RUM for prioritisation.

69.5 Alerting that isn't noise

ALERT ON SYMPTOMS, NOT CAUSES
   good:  "p99 latency > 2 s for 5 minutes"   (users are suffering)
   good:  "error rate > 1% for 3 minutes"
   poor:  "CPU > 80%"                          (so what? is anything broken?)
   poor:  "interface at 75%"                   (page a human at 3am for this?)

MAKE EVERY PAGE ACTIONABLE
   if the responder cannot do something about it right now, it is not a page.
   Everything else is a ticket or a dashboard.

USE MULTI-WINDOW BURN RATES (SRE style) rather than raw thresholds:
   page   if  error budget burns at 14.4x over 1 h  AND  6x over 5 min
   ticket if  it burns at 1x over 6 h
   -> fast burns page immediately; slow burns don't wake anyone

SUPPRESS DEPENDENT ALERTS
   if the core switch is down, do not send 400 alerts for what is behind it.
   Model dependencies, or alert on the root and summarise the rest.

ALWAYS ALERT ON:
   * certificate expiry (30/14/7 days) -- the most preventable outage there is
   * domain expiry
   * DNS resolution failures from an external vantage point
   * interface errors trending up (before they become loss)
   * conntrack / accept queue / socket exhaustion (Ch 66)

69.6 A minimal but complete stack

metrics        Prometheus (+ node_exporter, snmp_exporter, blackbox_exporter)
               or InfluxDB/Telegraf; Grafana for dashboards
flows          nfdump / GoFlow2 / ntopng, or cloud VPC flow logs -> object storage
synthetics     blackbox_exporter (ICMP/TCP/HTTP/DNS/TLS probes) + external provider
packets        tcpdump ring buffer on key hosts; a TAP/mirror to a capture box
                for critical segments
logs           syslog/journald -> Loki/Elasticsearch, correlated by timestamp
alerting       Alertmanager with dependency-aware routing and burn-rate rules

blackbox_exporter deserves a specific mention: one binary gives you ICMP, TCP connect, HTTP(S) with content and status assertions, TLS certificate expiry, and DNS checks — covering §69.2 almost entirely.

69.7 Real-world example

A company monitored their site with a single HTTP check from one cloud region. A BGP route leak (Ch. 50) made the site unreachable from three European ISPs for 90 minutes; the check — which ran from the same continent as the servers — stayed green throughout. Customers reported it. Adding probes from multiple external vantage points (and a RUM beacon) meant the next such event was detected in under two minutes by both the geographic spread of synthetic failures and a regional drop in real-user traffic.

Try it yourself

# a quick multi-layer probe of one target
H=example.com
ping -c 5 $H | tail -2
nc -zv -w3 $H 443
echo | openssl s_client -connect $H:443 -servername $H 2>/dev/null \
  | openssl x509 -noout -enddate                    # certificate expiry
curl -o /dev/null -s -w 'dns %{time_namelookup} tcp %{time_connect} tls %{time_appconnect} ttfb %{time_starttransfer} code %{http_code}\n' https://$H

# certificate expiry in days (put this in monitoring)
END=$(echo | openssl s_client -connect $H:443 -servername $H 2>/dev/null \
      | openssl x509 -noout -enddate | cut -d= -f2)
echo "$(( ( $(date -d "$END" +%s 2>/dev/null || date -jf '%b %d %T %Y %Z' "$END" +%s) - $(date +%s) ) / 86400 )) days left"