33. Nagle, delayed ACK, and the interaction bug

33.1 Nagle's algorithm (sender-side)

Goal: stop tiny "tinygram" segments (1-byte payload + 40-byte headers) from swamping the network (the 1980s "telnet over a congested link" problem).

Rule: if there is unacknowledged data outstanding, buffer new small writes until either (a) a full MSS worth accumulates, or (b) an ACK arrives. If nothing is unacked, send immediately.

Effect: back-to-back small write()s get coalesced into fewer, bigger segments — great for bulk/throughput, bad for latency-sensitive request/response.

Disable per-socket with TCP_NODELAY. Almost every RPC framework, database driver, game server, and low-latency system sets TCP_NODELAY. Redis, gRPC, nginx upstreams, PostgreSQL, SSH interactive — all disable Nagle.

33.2 Delayed ACK (receiver-side)

Goal: reduce pure-ACK traffic and enable piggybacking. Rule: don't ACK immediately; wait up to ~40 ms (Linux; 200 ms on some OSes) hoping to (a) have return data to piggyback the ACK on, or (b) receive a second segment so one ACK covers both. Standard says ACK at least every 2nd full segment.

33.3 The classic 40 ms (or 200 ms) stall

Nagle + delayed ACK can deadlock each other for a request/response pattern where the sender writes the request in two chunks (e.g. header then body):

Client (Nagle ON)                       Server (delayed ACK ON)
  write(header)  -> sent (nothing unacked yet)
  write(body)    -> BUFFERED by Nagle (header is unacked!)
                                         got header; no data to reply with yet;
                                         waits for a 2nd segment or 40ms before ACK
   ... both sides wait ...
                            (40 ms later) server delayed-ACK fires -> ACK header
  ACK arrives -> Nagle releases body -> sent
                                         server gets body, processes, replies

Every such round trip eats a fixed ~40 ms (or 200 ms on older stacks). Under load this shows up as throughput that's suspiciously close to N / 0.04 requests per second, or p50 latency pinned near 40/200 ms.

Fixes (any one):

• TCP_NODELAY on the client (most common; disables Nagle). Do this for virtually all interactive/RPC sockets.
• Write the whole request in one write() / use writev / buffer in userspace then flush once. No small-then-small pattern → Nagle never engages.
• TCP_QUICKACK on the server (Linux, must be re-set; it's not sticky) to suppress delayed ACK.
• TCP_CORK when you want coalescing deliberately (e.g. sendfile headers + file): cork, write parts, uncork to flush as full segments — controlled Nagle.

33.4 TCP_CORK vs TCP_NODELAY

• TCP_NODELAY = "never wait, send now" (latency).
• TCP_CORK = "hold everything until I uncork or 200 ms pass, then send full segments only" (efficiency, e.g. HTTP response headers + body in minimal packets). They're opposites; nginx toggles both around sendfile().

33.5 War story

A trading gateway's median order-ack latency was 40.1 ms in staging, sub-millisecond in production. Staging used a client library build without TCP_NODELAY; it wrote a 26-byte header then a variable body in a second send(). Nagle held the body until the server's delayed ACK fired 40 ms later. Setting TCP_NODELAY (one line) dropped median to 0.3 ms. Lesson: a latency floor at exactly 40 ms or 200 ms is Nagle × delayed-ACK until proven otherwise.

Try it yourself

strace -f -e trace=setsockopt yourapp 2>&1 | grep -E 'TCP_NODELAY|TCP_CORK|TCP_QUICKACK'
ss -tie dst <peer>          # 'ato:' is the delayed-ACK timeout in use
# Reproduce: nc without NODELAY vs a tiny python client that sets it:
python3 - <<'EOF'
import socket
s=socket.create_connection(("example.com",80))
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
EOF