Practice (25 min)

A. See the window in your own traffic:

sudo tcpdump -i any -n 'tcp port 443' -c 20

Look at the win value on each packet. After the handshake, the numbers are scaled — multiply by 2^(scale factor from the handshake) for the real value.

B. Look at live connection details (Linux — this is a great command):

ss -tin

Look for wscale:7,7 (window scaling active), rtt: (measured round-trip time), cwnd: (Chapter 25), and send X Mbps (its own throughput estimate).

C. Prove the window/RTT formula. Compare a download from something nearby with something far away:

curl -o /dev/null -s -w 'speed: %{speed_download} bytes/s   time: %{time_total}s\n' \
  https://speed.cloudflare.com/__down?bytes=10000000

Then compare ping times to servers at different distances. The relationship between latency and achievable throughput will be visible.

D. Check your system's buffer limits:

sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.core.rmem_max   # Linux
sysctl net.inet.tcp.recvspace net.inet.tcp.sendspace           # macOS

The third number in tcp_rmem is the maximum receive buffer. If it's smaller than your BDP, that's your speed limit.

Build it in Go (25 min) — drive the window to zero

Flow control is easiest to understand from the sender's side: when the receiver stops reading, the sender's Write stops returning. This lab connects a fast writer to a deliberately slow reader (4 KB every 500 ms) with small buffers, so the window fills within a second.

// zerowindow: a receiver that reads slowly, so you can watch flow control
// (Chapter 24) fill the receive buffer, shrink the window to zero, and make
// the sender's Write block.
//
//	go run ./zerowindow
//	# meanwhile:  ss -tnoi '( sport = :7071 or dport = :7071 )'   (Linux)
//	#             netstat -an -p tcp | grep 7071                   (macOS: watch Recv-Q / Send-Q)
package main

import (
	"fmt"
	"net"
	"time"
)

func main() {
	ln, err := net.Listen("tcp", "127.0.0.1:7071")
	if err != nil {
		panic(err)
	}

	go func() { // the SLOW receiver: 4 KB every 500 ms
		c, _ := ln.Accept()
		c.(*net.TCPConn).SetReadBuffer(64 * 1024) // a small window, so it fills fast
		buf := make([]byte, 4096)
		for {
			time.Sleep(500 * time.Millisecond)
			if _, err := c.Read(buf); err != nil {
				return
			}
		}
	}()

	c, err := net.Dial("tcp", ln.Addr().String())
	if err != nil {
		panic(err)
	}
	c.(*net.TCPConn).SetWriteBuffer(64 * 1024)

	chunk := make([]byte, 16*1024)
	total := 0
	for i := 0; i < 64; i++ {
		start := time.Now()
		n, err := c.Write(chunk) // returns once the bytes are in OUR send buffer
		if err != nil {
			fmt.Println(err)
			return
		}
		total += n
		if wait := time.Since(start); wait > 50*time.Millisecond {
			fmt.Printf("write #%-2d BLOCKED %-6v (total %4d KB) <- receiver's window is full\n",
				i, wait.Round(time.Millisecond), total/1024)
		} else {
			fmt.Printf("write #%-2d instant        (total %4d KB)\n", i, total/1024)
		}
	}
}

The program's own output shows writes going from instant to blocked. The kernel's view is the interesting part (Linux, during the run):

$ ss -tnoi '( sport = :7071 or dport = :7071 )'
ESTAB 79872  0        127.0.0.1:7071   127.0.0.1:51062           <- receiver: 78 KB unread (Recv-Q)
ESTAB 0      128000   127.0.0.1:51062  127.0.0.1:7071  timer:(persist,701ms,0)
     ... backoff:4 ... rwnd_limited:6008ms(100.0%) ... notsent:128000

How to read it:

• Receiver Recv-Q 79872: bytes the kernel holds that the application hasn't Read yet. The window it advertises shrinks by exactly this amount.
• Sender Send-Q 128000 / notsent:128000: bytes the sender has accepted from Write but can't send, because the window is closed.
• timer:(persist,...): the zero-window probe timer from this chapter. The sender periodically asks "is there room yet?", backing off each time (backoff:4).
• rwnd_limited:...(100.0%): Linux is telling you directly that the whole connection was limited by the receiver's window, not by the network. When you see this in production, look at the receiving application, not the network.

Exercises:

1. Change the reader's sleep to 0 and watch every write become instant. The network didn't change; the application did.
2. Remove the SetReadBuffer/SetWriteBuffer calls. Linux autotunes the buffers upwards, so it takes longer to block. How many KB get through before the first blocked write?
3. In a real service this looks like a consumer that's too slow, for example a log shipper behind a slow disk. Find the equivalent in the Linux guide's I/O stack chapter: a slow disk produces exactly this kind of backpressure.