24. The three-way handshake and connection setup

24.1 The exchange

Client (active open)                         Server (passive open, LISTENing)
  |                                            |
  |  1. SYN   seq=C_isn                         |
  |            win, MSS, wscale, SACK-OK, TS  ->|   creates a mini-state entry
  |                                            |   (SYN queue), replies:
  |                                            |
  |<- 2. SYN,ACK  seq=S_isn  ack=C_isn+1        |
  |               win, MSS, wscale, SACK-OK, TS |
  |                                            |
  |  3. ACK   seq=C_isn+1  ack=S_isn+1        ->|   moves conn: SYN queue ->
  |                                            |   accept queue; app accept()s
  |                                            |
  |========== ESTABLISHED both sides ==========|

Why three (not two)? Each side must (a) announce its ISN and (b) get it acknowledged, so both directions of the byte stream are synchronized. Two packets could sync only one direction; four would be redundant because the middle SYN+ACK combines the server's SYN with its ACK of the client's SYN.

24.2 The two queues on the server

SYN queue (a.k.a. half-open / tcp_max_syn_backlog): connections that got a SYN and sent a SYN-ACK, awaiting the final ACK.
Accept queue (a.k.a. completed / backlog, capped by somaxconn): handshake done, waiting for the app to accept().

Overflow behavior (Linux):

SYN queue full → new SYNs dropped (unless SYN cookies kick in).
Accept queue full → depending on net.ipv4.tcp_abort_on_overflow, the final ACK is ignored (client later retransmits it; connection completes if the app catches up) or the connection is RST. Silent 1-second connect stalls under load are almost always accept-queue overflow.

24.3 What each side allocates

A TCB (Transmission Control Block) per connection: the 4-tuple, both ISNs, current send/receive sequence numbers, windows, RTT estimators, congestion state, retransmit queue, timers, socket buffers. Tens of KB to hundreds of KB (dominated by socket buffers). A million connections is real but you must size tcp_mem, file descriptors, and RAM for it.

24.4 The Initial Sequence Number must be unpredictable

Original TCP used a simple clock-based ISN → attackers could guess it and inject data or spoof a whole connection (Morris, Mitnick). Modern stacks (RFC 6528) compute ISN = clock + hash(src IP, src port, dst IP, dst port, secret_key) — monotonic enough to avoid confusing old duplicate segments, random enough to resist off-path injection. (Off-path RST/data injection still matters — Ch. 52.)

24.5 SYN cookies — surviving a SYN flood

Under a SYN flood (Ch. 52), the SYN queue fills with half-open connections from spoofed sources and legit clients can't connect. SYN cookies (net.ipv4.tcp_syncookies=1, default) let the server keep no state for a half-open connection: it encodes the essential SYN info (MSS bucket, timestamp, tuple hash) into the SYN-ACK's ISN. If a real client returns the final ACK, the server reconstructs the connection from ack-1. Cost: a few TCP options can't be fully preserved (mitigated with the timestamp option). It only engages when the queue overflows.

24.6 TCP Fast Open (TFO)

Saves the handshake RTT for repeat connections: the first visit, the server sends a TFO cookie; on later visits the client puts SYN + cookie + request data in one packet and the server can process the request before the handshake finishes. Blocked or broken by some middleboxes/CDNs; used opportunistically.

24.7 Connection refused vs timeout vs unreachable — decode it

Client sees	Wire event	Usual cause
`Connection refused` (`ECONNREFUSED`)	RST in response to SYN	nothing listening on that port; host is up
`Connection timed out` (`ETIMEDOUT`)	SYNs sent, nothing back (SYN retries then give up, ~1–2 min)	firewall `DROP`, host down, black hole, wrong route, PMTU black hole on the SYN-ACK
`No route to host` (`EHOSTUNREACH`)	local routing failure or `ICMP host unreachable`	no route, ARP fail for gateway, link down
`Network is unreachable` (`ENETUNREACH`)	no matching route at all	missing default route
Connect succeeds then immediate RST	SYN-ACK then RST	app crashed on accept, backlog overflow with `abort_on_overflow`, or an IPS resetting

24.8 War story

A service saw ~0.1% of client connections take exactly 1.0 s or 3.0 s to establish, matching Linux's SYN/ACK retransmit backoff. ss -s showed ListenOverflows and ListenDrops climbing (nstat -az | grep -i listen). The app's listen() backlog was 128 while somaxconn was 4096 and traffic was bursty. Raising the app's backlog to 4096 and adding SO_REUSEPORT workers eliminated the stalls. Lesson: 1s/3s/7s connect latencies = SYN or final-ACK loss = check nstat for TcpExtListenOverflows/TcpExtListenDrops and the app's actual backlog argument.

Try it yourself

nstat -az | grep -Ei 'listen|syncookie|synretrans'
ss -ltn                                  # Recv-Q = current accept-queue depth, Send-Q = its max
sysctl net.ipv4.tcp_max_syn_backlog net.core.somaxconn net.ipv4.tcp_syncookies
sysctl net.ipv4.tcp_abort_on_overflow
# Watch a handshake:
sudo tcpdump -i any -n '(tcp[tcpflags] & (tcp-syn|tcp-rst)) != 0' -c20