Chapter 27 — Latency, bandwidth, and bufferbloat

The two numbers, and the analogy

   BANDWIDTH  = how many lanes the motorway has
                (bits per second: 100 Mbit/s, 1 Gbit/s)

   LATENCY    = how long the motorway is
                (milliseconds: 5 ms, 90 ms, 600 ms)

A truck full of hard drives has enormous bandwidth and terrible latency. A phone call has tiny bandwidth and needs low latency.

Upgrading bandwidth does nothing for latency. Going from 100 Mbit/s to 1 Gbit/s does not make a website load 10x faster, because most of a page load is round trips (Chapter 4), not bytes.

Where latency comes from

   total delay =  propagation  +  serialisation  +  queueing  +  processing

Physics floor: London to New York is ~5,600 km. Light in fibre does ~200 km/ms, so one way is ~28 ms, round trip ~56 ms minimum — and real paths aren't straight, so ~70–90 ms is typical. No amount of money makes that faster. Only moving the content closer does.

Bufferbloat: the one you can actually fix

Here's a problem you have probably experienced without knowing its name.

The symptom: when someone starts a big upload or download, everything else on the network becomes horrible. Video calls stutter, web pages crawl, games become unplayable — even though the download itself is going fine.

The cause: somewhere at the bottleneck (usually your home router or modem) there is a large, dumb buffer. TCP's congestion control (Chapter 25) increases speed until it sees loss — but with a huge buffer, packets aren't dropped, they're queued. So TCP keeps speeding up and fills the buffer, and now every packet — including your video call's, and your DNS lookups — has to wait behind a megabyte of bulk download.

   Idle:         ping = 20 ms
   During a big upload:  ping = 800 ms      <-- bufferbloat

How to test it (do this now — it takes 2 minutes):

# Terminal 1: watch latency continuously
ping 1.1.1.1

# Terminal 2: saturate your connection
curl -o /dev/null https://speed.cloudflare.com/__down?bytes=200000000
# and for the upload direction, use a speed test site

Watch the ping times in terminal 1. If they jump from 20 ms to hundreds of milliseconds, you have bufferbloat. Most home connections do.

Or use a browser test: waveform.com/tools/bufferbloat — it measures loaded vs unloaded latency and gives you a grade.

How to fix it: use Smart Queue Management (SQM) — a modern queueing algorithm (fq_codel or cake) at the bottleneck, which keeps the queue short and shares it fairly between flows.

• Many consumer routers have an SQM or "bufferbloat" setting — turn it on.
• OpenWrt has excellent SQM support.
• On a Linux router: tc qdisc replace dev <wan> root cake bandwidth 90Mbit (set slightly below your real line rate, so the queue forms in the smart device rather than the dumb modem).

You typically give up ~5–10% of peak throughput and get a 10x improvement in latency under load. It's the single best network upgrade most people can make, and it's free.

ping -c 10 <your router>          # ~1 ms
ping -c 10 1.1.1.1                 # your ISP + a bit
ping -c 10 <something far away>    # distance

curl -w 'dns %{time_namelookup}  tcp %{time_connect}  tls %{time_appconnect}  ttfb %{time_starttransfer}  total %{time_total}\n' \
     -o /dev/null -s https://example.com

tc qdisc show