Chapter 59 — Network performance engineering: QoS, packet loss, shaping, saturation

Build it in Go (30 min) — measure the bandwidth-delay product

This lab sends data as fast as possible to a sink and reports throughput. It can pin the socket buffer (SetWriteBuffer, which disables Linux's autotuning) and open parallel streams, which lets you reproduce the first law on demand:

// bdp: feel the bandwidth-delay product (Chapters 24 and 59).
//
// One TCP connection can have at most one window of data "in flight". If the
// window (bounded by socket buffers) is smaller than bandwidth x RTT, the pipe
// is never full and throughput = window / RTT, no matter how fast the link is.
//
//	go run ./bdp -serve :7500
//	go run ./bdp -to HOST:7500 -buf 65536            # small buffer
//	go run ./bdp -to HOST:7500                       # kernel autotuning
//	go run ./bdp -to HOST:7500 -streams 4 -buf 65536 # parallel streams
//
// Add delay first to make the effect obvious (Linux, on the client):
//
//	sudo tc qdisc add dev eth0 root netem delay 50ms
package main

import (
	"flag"
	"fmt"
	"io"
	"net"
	"os"
	"sync"
	"sync/atomic"
	"time"
)

func main() {
	serve := flag.String("serve", "", "run the sink server on this address")
	to := flag.String("to", "", "server to send to")
	buf := flag.Int("buf", 0, "socket send buffer in bytes (0 = kernel autotuning)")
	streams := flag.Int("streams", 1, "parallel connections")
	secs := flag.Int("secs", 5, "test duration in seconds")
	flag.Parse()

	if *serve != "" {
		ln, err := net.Listen("tcp", *serve)
		if err != nil {
			fmt.Println(err)
			os.Exit(1)
		}
		fmt.Println("sink listening on", ln.Addr())
		for {
			c, err := ln.Accept()
			if err != nil {
				continue
			}
			go func() { io.Copy(io.Discard, c); c.Close() }()
		}
	}
	if *to == "" {
		flag.Usage()
		os.Exit(2)
	}

	rtt := measureRTT(*to)
	var total atomic.Int64
	deadline := time.Now().Add(time.Duration(*secs) * time.Second)
	var wg sync.WaitGroup
	for i := 0; i < *streams; i++ {
		c, err := net.Dial("tcp", *to)
		if err != nil {
			fmt.Println(err)
			os.Exit(1)
		}
		if *buf > 0 {
			c.(*net.TCPConn).SetWriteBuffer(*buf) // pins the size: disables autotuning
		}
		wg.Add(1)
		go func() {
			defer wg.Done()
			defer c.Close()
			chunk := make([]byte, 128*1024)
			for time.Now().Before(deadline) {
				n, err := c.Write(chunk)
				total.Add(int64(n))
				if err != nil {
					return
				}
			}
		}()
	}
	wg.Wait()

	mbps := float64(total.Load()) * 8 / float64(*secs) / 1e6
	fmt.Printf("streams=%d buf=%d rtt~%v  ->  %.1f Mbit/s\n", *streams, *buf, rtt.Round(time.Millisecond), mbps)
	if *buf > 0 && rtt > 0 {
		// Linux doubles the SO_SNDBUF you ask for (half is bookkeeping overhead),
		// so roughly 2*buf bytes can be in flight per connection.
		ceiling := float64(2**buf) * 8 / rtt.Seconds() / 1e6 * float64(*streams)
		fmt.Printf("theory: streams x 2*buf / RTT ~ %.1f Mbit/s\n", ceiling)
	}
}

// measureRTT takes the fastest of three TCP handshakes: one handshake ~ 1 RTT.
func measureRTT(addr string) time.Duration {
	best := time.Hour
	for i := 0; i < 3; i++ {
		start := time.Now()
		c, err := net.Dial("tcp", addr)
		if err != nil {
			fmt.Println(err)
			os.Exit(1)
		}
		best = min(best, time.Since(start))
		c.Close()
	}
	return best
}

Real results between two containers, with 50 ms of delay added by netem:

$ go run ./bdp -to sink:7500 -buf 65536
streams=1 buf=65536 rtt~55ms  ->  18.4 Mbit/s
theory: streams x 2*buf / RTT ~ 19.0 Mbit/s

$ go run ./bdp -to sink:7500                    # kernel autotuning
streams=1 buf=0 rtt~52ms  ->  230.7 Mbit/s

$ go run ./bdp -to sink:7500 -buf 65536 -streams 4
streams=4 buf=65536 rtt~51ms  ->  72.9 Mbit/s
theory: streams x 2*buf / RTT ~ 82.0 Mbit/s

What to notice:

• The theory line predicts the measurement to within a few percent. The network was the same in all three runs. Only the window changed.
• Linux doubles the SO_SNDBUF you ask for (half is bookkeeping), so 2 × buf is in flight. This surprises everyone who reads man 7 socket for the first time.
• Setting a buffer size disables autotuning. Autotuning reached 230 Mbit/s without help. Hard-coding "big" buffers in an application often makes things worse than leaving the kernel alone. Tune the kernel's ceilings (net.ipv4.tcp_wmem/tcp_rmem max) for long fat networks, not each socket.
• Parallel streams multiply the window. That's why download managers, aws s3 cp, and rclone open many connections on high-RTT paths.

Exercises:

1. Add -delay values of 10, 50, and 150 ms (with tc netem) and plot throughput for the pinned 64 KB buffer. It should fall as 1/RTT.
2. Add 1% loss and compare CUBIC with BBR by switching sudo sysctl -w net.ipv4.tcp_congestion_control=bbr on the sender between runs (load it first with sudo modprobe tcp_bbr if needed). Which degrades less? Compare both with Mathis' prediction above.
3. Run it across the internet between two cloud regions. How close does a single stream get to the instance's advertised bandwidth?