30. Flow control: the sliding window, window scaling, and SWS

30.1 The sliding window

The receiver advertises rwnd in every ACK = free space in its socket receive buffer. The sender may send up to snd_una + min(rwnd, cwnd) — i.e. bytes from "oldest unacked" forward. As ACKs come in, the window slides right.

   sent+acked | sent, not yet acked |  may send now  | can't send yet
 --------------+---------------------+----------------+---------------->
             snd_una              snd_nxt        snd_una + wnd
              <-------- window (min(rwnd,cwnd)) -------->

If the app on the receiver stops calling read(), its buffer fills, rwnd shrinks toward 0, and the sender throttles — automatically. This is why a slow consumer naturally slows a fast producer over TCP ("backpressure for free").

30.2 Window scaling — the 64 KB wall

The header's Window field is 16 bits → max 65 535 bytes. Max throughput is window / RTT:

Window Scale option (SYN only): a shift count s (0–14); the real window is advertised << s. With s=7, a 65 535 advertisement means ~8 MB. Both SYNs must carry it or scaling is off for the whole connection. Linux enables it by default (net.ipv4.tcp_window_scaling=1); the actual max is bounded by net.ipv4.tcp_rmem / tcp_wmem and autotuning (Ch. 35).

Middleboxes that strip or rewrite the WScale option cause a nasty asymmetry: one side scales, the other doesn't, and windows are misinterpreted by orders of magnitude → stalls or resets.

30.3 Zero window and the persist timer

If the receiver's buffer is completely full it advertises window = 0. The sender stops. When the app drains the buffer, the receiver sends a window update ACK. But ACKs are not retransmitted — if that update is lost, both sides wait forever. The persist timer prevents deadlock: the sender periodically sends a 1-byte zero-window probe; the receiver responds with the current (possibly still 0) window. Eventually the real window is learned.

Seeing frequent TCP ZeroWindow / TCP Window Full in Wireshark = the receiving application is too slow (not reading fast enough). The network is fine; fix the consumer (more worker threads, faster processing, bigger buffer).

30.4 Silly Window Syndrome (SWS)

Pathology where the window is repeatedly advanced by tiny amounts, so the connection exchanges many small segments (huge header overhead). Two guards:

• Receiver side (David Clark's fix): don't advertise a larger window until it can grow by at least one MSS (or half the buffer).
• Sender side (Nagle's algorithm, Ch. 33): don't send a small segment while a previous small segment is still unacked; coalesce.

30.5 Buffer autotuning

Modern Linux dynamically sizes each connection's receive buffer between tcp_rmem min/default/max based on the measured bandwidth-delay product (tcp_moderate_rcvbuf=1). You usually shouldn't set SO_RCVBUF/SO_SNDBUF manually — doing so disables autotuning and often hurts. Only raise the tcp_rmem/tcp_wmem max (Ch. 35, 37) for long-fat-network workloads.

30.6 War story

A log shipper's throughput to a central collector was stuck at ~90 Mbit/s despite a 10G network, RTT ~25 ms. ss -tim showed rcv_space/rwnd pinned near 3 MB — the collector app had called setsockopt(SO_RCVBUF, 2MB) "to bound memory," which capped and froze the window and disabled autotuning. 3MB / 0.025s ≈ 960 Mbit/s ceiling, minus overhead ≈ observed. Removing the manual SO_RCVBUF let autotuning grow it to ~25 MB and throughput hit multi-Gbit/s. Lesson: manual socket buffer sizes usually do more harm than good; tune the sysctl ceilings and let autotuning work.

Try it yourself

ss -tim dst <peer>       # rcv_space, snd_wnd, rcv_wnd, mss, cwnd, bytes_acked...
sysctl net.ipv4.tcp_window_scaling net.ipv4.tcp_moderate_rcvbuf
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem
nstat -az | grep -Ei 'TCPWantZeroWindow|TCPZeroWindow|TCPBacklogDrop|PruneCalled'
# Wireshark filter: tcp.analysis.zero_window || tcp.analysis.window_full