35. Bandwidth-delay product and buffer sizing

35.1 The BDP

BDP (bytes) = bottleneck_bandwidth (bytes/s) x round_trip_time (s)

It's the amount of data "in the pipe" when the pipe is full. To keep a link 100% utilized, the sender must be allowed to have at least one BDP outstanding (unacked) — so min(cwnd, rwnd, sndbuf) ≥ BDP.

35.2 Turning BDP into settings

For a single flow to fill a cross-country 1 Gbit/s path you need ~9 MB of window. That requires:

1. Window scaling on (Ch. 30) — 16-bit window alone maxes at 64 KB.
2. Receive buffer max ≥ BDP: net.ipv4.tcp_rmem third value ≥ ~16 MB (headroom over 9 MB; autotuning also reserves ~50% for metadata/overhead — the effective window is roughly half of SO_RCVBUF).
3. Send buffer max ≥ BDP: net.ipv4.tcp_wmem third value similarly, plus net.core.wmem_max.
4. Enough cwnd — congestion control must be allowed to grow there, i.e. low enough loss (√p term, Ch. 36) or a model-based CC (BBR).

Rule of thumb for a WAN transfer host: set tcp_rmem/tcp_wmem max to 2 × BDP_max for your worst path, and net.core.rmem_max/wmem_max to match. Don't set SO_RCVBUF/SO_SNDBUF in the app — that pins the value and disables autotuning.

35.3 Buffers that are too big: bufferbloat

The opposite failure. An over-large buffer at the bottleneck (home router, cable modem, VM host, a qdisc with a 1000-packet FIFO) lets a loss-based CC (CUBIC) fill it with a full BDP+ of standing queue. Throughput stays high but latency explodes — a full 1 s of buffering on a slow uplink. Every other flow sharing that link (your video call, DNS, ACKs) now waits behind the bulk transfer. Signature: ping RTT jumps from 20 ms to 200–1000 ms only while a big upload/download runs.

Fixes:

• AQM (Active Queue Management) at the bottleneck: fq_codel (default qdisc on modern Linux) or CAKE — they keep the queue short by dropping/ECN- marking early and isolate flows so a bulk transfer can't starve a sparse one.
• BBR on the senders (paces to BtlBw, keeps ~0 standing queue).
• On a Linux router/gateway: tc qdisc replace dev <wan> root cake bandwidth <slightly-below-line-rate> (shaping just under line rate moves the queue from the dumb modem into the smart Linux qdisc).

35.4 The "50%" gotchas

• Linux reports SO_RCVBUF/SO_SNDBUF as double what you set (setsockopt(...,X) → getsockopt returns 2X); the extra is bookkeeping reserve. Effective window ≈ SO_RCVBUF / 2.
• net.ipv4.tcp_adv_win_scale (default 1 on older, -2 on newer kernels) controls that application-vs-window split.

35.5 War story

A media company's transcode farm pulled 4 GB masters from an object store cross-region and each pull took ~9 minutes (~60 Mbit/s) on a 10 Gbit/s network, RTT 32 ms. BDP ≈ 40 MB. tcp_rmem max was the distro default 6 MB → effective window ~3 MB → 3MB / 0.032s ≈ 750 Mbit/s theoretical, less in practice. Raising tcp_rmem/tcp_wmem max to 128 MB and core.rmem_max/ wmem_max to 128 MB took a single stream to ~6 Gbit/s (pull time ~1 min). Lesson: for WAN bulk transfer, if buffer_max < 2×BDP you are buffer-bound no matter the link speed. Compute BDP first.

Try it yourself

# Measure BDP inputs:
ping -c20 <peer> | tail -1                     # avg RTT
iperf3 -c <peer> -t20                          # throughput (single stream)
iperf3 -c <peer> -t20 -P8                      # with 8 parallel streams (window/CC limited?)
# Current ceilings:
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.core.rmem_max net.core.wmem_max
ss -tim dst <peer>                             # watch rcv_space / snd_wnd grow
tc -s qdisc show dev eth0                      # is fq_codel/fq/cake active? drops?