Part 11 — Watching your network (monitoring)

Part 8 was about investigating a problem you already knew about. This part is about finding out first.

Chapter 45 — What to measure, and why averages lie

In one sentence

Measure what users feel (latency, errors, throughput) and what predicts failure (errors, drops, saturation) — and always look at percentiles, never averages.

The four kinds of signal

   1. DEVICE / LINK METRICS    "is the pipe healthy?"
      utilisation, errors, drops, interface up/down
      -> cheap, constant, from SNMP or kernel counters

   2. FLOW RECORDS             "who is talking to whom, and how much?"
      summaries of each conversation
      -> answers "what is using my bandwidth?"

   3. ACTIVE PROBES            "what would a user experience right now?"
      ping, TCP connect, HTTP checks, DNS lookups
      -> works even when there is no real traffic (3am outages)

   4. PACKET CAPTURE           "what EXACTLY happened?"
      -> expensive; on demand or in a small rolling buffer

Cost rises as you go down; precision rises with it. Run 1 and 3 always, 2 sampled, 4 when you need it.

The metrics that catch real incidents

On a link or interface:

Metric	Alert when
Utilisation (in/out)	Above ~70–80% sustained — queueing starts long before 100%
Errors / CRC	Any sustained increase. This is a failing cable or optic.
Discards / drops	Rising — buffers are overflowing
Interface flaps	More than once an hour

On a Linux host (these are the ones people forget):

nstat -az | grep -Ei 'ListenOverflows|ListenDrops|TCPBacklogDrop|RetransSegs'

Counter	Means
`TcpExtListenOverflows`	The accept queue is full — connections are being dropped before your app sees them
`TcpRetransSegs` / `TcpOutSegs`	Ratio ≈ your packet loss rate
`UdpRcvbufErrors`	UDP packets dropped because the app read too slowly

This is the most valuable point in the chapter: connection-level failures are invisible in application logs, because the connection never became a request. If you don't scrape kernel counters, you will never see them.

On a service: request rate, error rate (4xx and 5xx counted separately — Chapter 28), latency percentiles, and the timing breakdown from Chapter 4 (DNS / TCP / TLS / first byte).

Why averages lie

   "Average latency: 50 ms"        sounds completely fine.

   Reality:
      p50 (half of users)   =    20 ms
      p95                    =   180 ms
      p99 (1 in 100)         = 4,000 ms   <-- these people are furious

   The average hid the entire problem.

Rules:

Always chart p50, p95, p99.
Never average a percentile across servers or time buckets — it's mathematically meaningless. Aggregate from histograms.
For saturation, plot the maximum. A five-minute average hides a ten-second stall completely.

Watch for round numbers

Chapter 26 taught you this and it applies to monitoring too. If a latency chart has a hard floor or a cluster at a suspiciously round value, it's a timer, not physics:

   40 ms    Nagle + delayed ACK      (Chapter 26)
   200 ms   TCP retransmission timer (Chapter 22)
   1/3/7 s  SYN retries              (Chapter 21)
   5 s      a dead DNS server        (Chapter 18)
   60 s     an idle timeout          (Chapters 23, 43)

Practice (20 min)

# the kernel counters nobody watches
nstat -az | grep -Ei 'ListenOverflows|ListenDrops|RetransSegs|OutSegs|UdpRcvbufErrors'

# interface errors (should be zero and STAY zero)
ip -s link show eth0
ethtool -S eth0 2>/dev/null | grep -Ei 'err|drop|crc'

# take two samples 10 seconds apart -- rates matter, not totals
nstat -n; sleep 10; nstat

# per-connection health (Chapter 33)
ss -ti state established | grep -oE 'rtt:[0-9.]+|retrans:[0-9]+/[0-9]+' | head

Write your baseline numbers into notes.md. You cannot spot an anomaly without knowing what normal looks like.

Check yourself

Name the four kinds of monitoring signal.
Why can't application logs show you accept-queue overflows?
Why is an average latency figure misleading?
You see latency clustered at exactly 200 ms. What does that suggest?