44. Diagnosing latency, loss, throughput, and bufferbloat

44.1 The four questions, in order

When "the network is slow," decide which of these it is — the fixes are totally different:

1. Latency (RTT too high or too variable) — pages feel sluggish, RPC p99 bad, throughput of short transfers poor.
2. Loss (packets dropped) — throughput of bulk transfers poor, retransmit counters climbing, video pixelation.
3. Throughput ceiling with no loss — bulk transfer plateaus well below link rate; usually window/BDP/CC-limited (Ch. 35–36) or app-limited.
4. Bufferbloat — throughput is fine but latency balloons while a transfer runs, wrecking everything else on the link.

44.2 Step-by-step triage

A. Is it the network at all?
   - ss -tie: is the socket `app_limited`? busy: low? -> it's the app, not the net.
   - Capture: long gap before server PSH, no missing packets -> server think-time.
   - Compare curl time_starttransfer vs time_total -> TTFB (server) vs body (transfer).

B. Measure baseline latency (idle):
   - ping / mtr to the ENDPOINT (not a router). Note min / avg / mdev.
   - `ss -tie` minrtt vs rtt: rtt >> minrtt means standing queue somewhere.

C. Measure latency UNDER LOAD (the bufferbloat test):
   - Start `mtr <peer>` or `ping <peer>` in one window.
   - In another, run a big transfer both directions:
       iperf3 -c <peer> -t30            (upload)
       iperf3 -c <peer> -t30 -R         (download)
   - If ping RTT jumps 5-50x during the transfer -> BUFFERBLOAT at the bottleneck.
   - Which direction spikes tells you which buffer (up = your uplink/modem).

D. Quantify loss:
   - mtr -rwzbc300 : loss% that PERSISTS to the final hop is real.
   - nstat delta:  TcpRetransSegs / TcpOutSegs  ~= loss rate (sender side).
   - Two-ended capture: segments present at sender, absent at receiver.

E. Throughput ceiling with no loss:
   - 1 stream vs `iperf3 -P8`: if 8x faster -> window/CC/app-limited, not link.
   - ss -tie: cwnd small & flat, app_limited absent, retrans 0 -> rwnd/buffer cap.
   - Check window scaling on both SYNs; raise tcp_rmem/wmem max to 2*BDP.

44.3 Bufferbloat, in depth

Cause: a device at the bottleneck (home router, DSL/cable modem, LTE modem, VM host vNIC, a pfifo qdisc with txqueuelen 1000) has a large dumb FIFO. A loss-based sender (CUBIC) keeps increasing cwnd until that buffer overflows; in steady state it stays nearly full. A full 1 MB buffer draining at 8 Mbit/s = 1000 ms of added one-way delay for every packet, including your DNS, your SSH keystrokes, your video call, and the ACKs of the very transfer causing it.

Diagnosis: RTT idle 15 ms → RTT during upload 600 ms. ss -tie shows rtt:600 minrtt:15. tc -s qdisc shows a pfifo_fast/fifo with a big backlog and no drops until overflow.

Fixes (in order of leverage):

1. AQM at the bottleneck. On a Linux gateway: tc qdisc replace dev <wan> root cake bandwidth 19Mbit (set ~90–95% of true line rate so the queue forms in CAKE, which manages it, not in the modem). Or fq_codel if cake unavailable. Consumer routers: enable "Smart Queue Management / SQM" (OpenWrt sqm-scripts).
2. fq_codel as the host default qdisc (net.core.default_qdisc=fq_codel) — helps a host that is itself the bottleneck, and isolates flows (a bulk flow can't delay a sparse flow's packets).
3. BBR on your senders — paces to estimated bandwidth, keeps standing queue near zero.
4. Reduce txqueuelen on the egress interface (e.g. ip link set eth0 txqueuelen 100) and NIC ring buffers if absurdly large.
5. ECN end-to-end so the AQM can mark instead of drop.

44.4 Latency decomposition

Total one-way delay = propagation + serialization + queueing + processing

44.5 App-limited vs network-limited (don't fix the wrong thing)

ss -tie reports app_limited when the socket could have sent more but the application didn't write() fast enough. If you see it, no amount of buffer/CC tuning helps — the bottleneck is your code (single-threaded producer, small writes, blocking disk read, GC pauses, Content-Length computed by buffering the whole body first). busy: time near the total connection time = network-limited; much lower = app-limited.

44.6 War story

A CI system's artifact downloads were "slow" (3 min for 500 MB, ~22 Mbit/s) on a "100 Mbit/s" office link. Idle ping to the artifact host: 12 ms. Ping during download: 340 ms, and every other user's web browsing crawled. Classic bufferbloat in the office's cable modem. The office gateway (a Linux box) got tc qdisc replace dev wan0 root cake bandwidth 95Mbit; download rose to ~90 Mbit/s and in-download ping stayed at ~20 ms. Lesson: "slow downloads + everyone else's internet dies during them" = bufferbloat; the cure is AQM/shaping at the bottleneck, not a bigger pipe.

Try it yourself

# The two-terminal bufferbloat test:
#   term1:
mtr -o "L S  A M" <peer>
#   term2:
iperf3 -c <peer> -t30 ; iperf3 -c <peer> -t30 -R
# Or a one-shot score:
# (browser) https://www.waveform.com/tools/bufferbloat  or  fast.com (shows loaded latency)
ss -tie dst <peer> | grep -oE 'minrtt:[0-9.]+|rtt:[0-9.]+/[0-9.]+|app_limited|busy:[0-9]+ms'
tc -s qdisc show dev eth0