36. Throughput math: loss, latency, and the Mathis equation

36.1 The equations every network engineer should know

Window-limited throughput (no loss):

Throughput ~ Window / RTT

Loss-limited throughput of a single loss-based TCP flow (Mathis et al., 1997):

Throughput ~ (MSS / RTT) x (C / sqrt(p))

  p = packet loss probability
  C ~ 1.22  (for the standard AIMD sawtooth with periodic loss)

A more complete form (Padhye/PFTK, 1998) adds the RTO term for heavier loss:

Throughput ~ MSS /
   ( RTT*sqrt(2p/3)  +  RTO*min(1, 3*sqrt(3p/8))*p*(1 + 32p^2) )

36.2 What Mathis tells you (the intuition)

• Throughput is inversely proportional to RTT — double the latency, halve the speed.
• Throughput is inversely proportional to √loss — going from 0.01% to 1% loss (100×) cuts throughput 10×.
• A loss-based flow cannot sustain a large window on a path with even modest loss, regardless of link capacity.

Worked example — 1 Gbit/s link, RTT 80 ms, MSS 1448:

This is why a "1 Gbit/s" trans-continental link delivers single-digit Mbit/s per CUBIC stream once there's any real loss — and why the workarounds are: many parallel streams (N streams ≈ N× throughput until they collectively cause loss), BBR (models bandwidth, ignores that loss), FEC/erasure coding, or fixing the loss.

36.3 Parallel streams

N independent flows sharing a path get roughly N × a single flow's Mathis rate — until their combined load pushes p up. Tools: iperf3 -P, aria2c -x, GridFTP, rclone --multi-thread-streams, Aspera/UDT (UDP-based, bypasses the √p problem entirely). This is also why a browser's 6 connections and HTTP/2's multiplexing-over-one-connection behave differently under loss (HTTP/2's single TCP connection suffers head-of-line blocking; HTTP/3 fixes it — Ch. 39).

36.4 Goodput vs throughput

• Throughput: bytes on the wire per second (includes headers, retransmits).
• Goodput: useful application bytes per second delivered.
• Overhead sources: 40–52 B headers per ~1448 B (~3.5%), retransmissions (2×p roughly), TLS record overhead (~0.5–1%), slow-start ramp (matters for short transfers), and connection setup RTTs.
• For a short transfer (e.g. 50 KB HTTP response), you never leave slow start; throughput math is irrelevant and round trips dominate: time ≈ (handshake RTTs) + ceil(log2(response/IW)) × RTT. Optimize RTT count, not bandwidth.

36.5 War story

A genomics lab moved ~2 TB/day between universities over a shared 10 Gbit/s research link, RTT 55 ms, measured loss ~0.02%. Single scp ≈ 15 Mbit/s (Mathis predicts ~12–20). They blamed the link. Solution stack: (1) tcp_rmem/wmem max → 256 MB, (2) switch senders to BBR + fq, (3) use a parallel tool (bbcp/GridFTP, 16 streams). Result: ~8 Gbit/s aggregate, transfer window shrank from all day to ~45 min. Lesson: don't argue with MSS/(RTT·√p); engineer around it with buffers + BBR + parallelism.

Try it yourself

# Estimate p from counters over a known interval:
nstat -az | grep -E 'TcpRetransSegs|TcpOutSegs'   # p ~ RetransSegs / OutSegs (rough)
# Compare 1 vs N streams (if Nx faster, you're loss/CC-limited, not link-limited):
iperf3 -c <peer> -t30 ; iperf3 -c <peer> -t30 -P16
# Compare CUBIC vs BBR on the same lossy path:
iperf3 -c <peer> -t30 -C cubic ; iperf3 -c <peer> -t30 -C bbr