Part XX — Monitoring and telemetry

Parts XII–XIII covered investigating a problem you already know about. This part is about knowing first — what to measure continuously, with what mechanism, and how to turn it into alerts that aren't noise.

66. What to measure: the four signal families

66.1 The taxonomy

+-------------------+---------------------------+--------------------------+
| FAMILY            | ANSWERS                   | MECHANISM                |
+-------------------+---------------------------+--------------------------+
| 1. DEVICE/LINK    | "is the pipe healthy?"    | SNMP, kernel counters,   |
|    metrics        | utilisation, errors,      | ethtool, /proc, gNMI     |
|                   | drops, temperature        |                          |
+-------------------+---------------------------+--------------------------+
| 2. FLOW records   | "who talked to whom,      | NetFlow, IPFIX, sFlow,   |
|                   | how much, over what?"     | conntrack, eBPF          |
+-------------------+---------------------------+--------------------------+
| 3. ACTIVE probes  | "what does a user         | ping, TWAMP, synthetic   |
|    (synthetic)    | actually experience?"     | HTTP checks, traceroute  |
+-------------------+---------------------------+--------------------------+
| 4. PACKETS        | "what EXACTLY happened?"  | tcpdump, mirrors/TAPs,   |
|                   |                           | continuous PCAP          |
+-------------------+---------------------------+--------------------------+

Cost and volume rise sharply down the list; specificity rises with it. A mature setup runs 1 and 3 always, 2 sampled always, and 4 on demand or in a short ring buffer.

66.2 The signals that actually catch incidents

Per interface (from Chapter 48's drop table):

Metric	Why	Alert on
`ifInOctets` / `ifOutOctets`	Utilisation	> 70–80% sustained (queueing starts well before 100%)
`ifInErrors` / CRC errors	Physical layer failing	any sustained increase
`ifInDiscards` / `ifOutDiscards`	Buffer exhaustion / policing	rate of change
Interface flaps (`ifOperStatus`)	Bad optic/cable/power	> 1 flap in an hour
Optical Tx/Rx power (DOM/DDM)	Failing SFP or dirty fibre	outside vendor range, or trending down
Queue drops per class	QoS starvation (Ch. 51)	any drop in the priority queue

Per host (Linux, Ch. 43/48):

nstat / /proc/net/snmp   TcpRetransSegs / TcpOutSegs      -> loss proxy
                         TcpExtListenOverflows/Drops       -> accept queue full  (Ch 24)
                         TcpExtTCPBacklogDrop              -> socket backlog full
                         UdpRcvbufErrors                   -> UDP receive drops   (Ch 22)
                         IpReasmFails                      -> fragment problems   (Ch 15)
/proc/net/softnet_stat   col2 drops, col3 time_squeeze     -> softirq overload    (Ch 48)
conntrack -S             insert_failed, drop               -> state table full    (Ch 53)
ss -ti                   rtt, retrans, cwnd per socket     -> per-connection health

Per service (the ones users feel): TTFT/TTFB, request rate, error rate (4xx vs 5xx separately), p50/p95/p99 latency, connection setup time, TLS handshake time, DNS resolution time.

66.3 RED, USE, and the golden signals

Three complementary frames, all useful:

Frame	Measures	Best for
USE (Brendan Gregg)	Utilisation, Saturation, Errors — per resource	Links, NICs, queues, CPUs
RED	Rate, Errors, Duration — per service	Request-driven services
Four golden signals (Google SRE)	Latency, Traffic, Errors, Saturation	Anything user-facing

Saturation is the one people forget, and it's the one that predicts trouble. A link at 60% utilisation with a full queue is already hurting; utilisation alone hides it. Measure queue depth and drops, not just throughput.

66.4 Percentiles, not averages

Average latency 50 ms sounds fine.
   p50 = 20 ms, p99 = 4,000 ms  -> 1% of users are having a terrible time.

Rules:
  * always chart p50, p95, p99 (and p99.9 if you have the volume)
  * NEVER average percentiles across instances or time buckets -- it is
    mathematically meaningless. Aggregate from histograms instead.
  * plot MAX for saturation metrics; a 5-minute average hides a 10-second stall

Round-number latency clusters are timers, not physics — 40 ms (Nagle/delayed ACK, Ch. 33), 200 ms (RTO, Ch. 28), 1/3/7 s (SYN retries, Ch. 24), 5 s (dead DNS server, Ch. 20).

66.5 Real-world example

A payments service alerted only on CPU and HTTP 5xx rate. Every few days it saw a 30-second spike of connection failures that never triggered anything, because the failures happened before the request reached the application — the accept queue was overflowing (TcpExtListenOverflows) during traffic bursts. Nothing in the app logs, nothing in CPU. Adding nstat counters to the metrics pipeline surfaced it in a day; the fix was a bigger listen() backlog. Connection-level failures are invisible to application logs — you must scrape kernel counters.

Try it yourself

nstat -az | grep -Ei 'ListenOverflows|ListenDrops|TCPBacklogDrop|RetransSegs|OutSegs'
cat /proc/net/softnet_stat | head -4     # col2=drops col3=time_squeeze (hex)
ip -s link show eth0                      # RX/TX errors, dropped
ethtool -S eth0 | grep -Ei 'err|drop|crc|miss'
ss -ti state established | grep -oE 'rtt:[0-9.]+|retrans:[0-9]+/[0-9]+' | head