Part 6 — Layer 4: Transport

How it works

1. TCP vs UDP.

"TCP is a byte stream" means: if you send() 100 bytes twice, the other side may recv() 200 bytes once, or 37 then 163. Every TCP protocol must frame its own messages (HTTP uses Content-Length / chunked encoding; gRPC uses a length prefix). Forgetting this is a classic bug in hand-written protocols.

2. The TCP header and the fields that matter:

Src port 54012 | Dst port 443
Sequence number   — byte offset of this segment's first byte in the stream
Ack number        — "I've received everything up to byte N-1, send N next"
Flags             — SYN, ACK, FIN (graceful close), RST (abort), PSH, ECE/CWR (ECN)
Window            — receive window: how many more bytes I can buffer (flow control)
Options           — MSS, Window Scale, SACK, Timestamps

3. Reliability. Every byte has a sequence number. The receiver ACKs what it has received. If an ACK doesn't arrive within the retransmission timeout (RTO), or the sender sees three duplicate ACKs (fast retransmit), it sends the data again. Linux's minimum RTO is 200 ms, and the initial RTO before any RTT measurement (the SYN) is 1 second. So:

One lost packet in the wrong place costs 200 ms to 1 s or more. A lost SYN adds a full second to connection setup. This is why 0.1% packet loss damages p99 latency far more than you'd expect.

4. Flow control vs congestion control. People mix these up:

Slow start: a new connection starts with cwnd = 10 segments (~14 KB) and roughly doubles every RTT until it sees loss (CUBIC, the Linux default) or until it models the path's bandwidth (BBR). Consequence:

Transferring 200 KB on a fresh connection, RTT 40 ms, ignoring loss:
  RTT 1: 14 KB    RTT 2: 28 KB    RTT 3: 56 KB    RTT 4: 112 KB   → ~4 RTTs ≈ 160 ms
On a warmed-up, reused connection: ~1 RTT ≈ 40 ms

5. Bandwidth-delay product (BDP). To fill a pipe you need bandwidth × RTT bytes in flight. 1 Gb/s × 100 ms = 12.5 MB. If socket buffers or windows are smaller than that, a single connection can't use the bandwidth you're paying for. That's why cross-region bulk copies use parallel streams.

6. Closing. A graceful close is FIN → ACK → FIN → ACK. The side that closes first enters TIME_WAIT for 2×MSL (60 s on Linux) to absorb stray packets. A busy client that opens and closes thousands of short connections can run out of ephemeral ports (Linux default range 32768–60999, about 28k) because of TIME_WAIT. RST is an abort: "this connection doesn't exist" or "I'm giving up". It's what you see as ECONNRESET / "connection reset by peer".

7. Ports and sockets in the kernel. A server listen()s on :8080. The kernel completes the handshake without involving your app and queues the connection in the accept backlog. Your app calls accept() to get it. If your app is too slow to accept, the backlog fills and new SYNs get dropped. To clients that looks like a 1–3 s connect delay.

8. L4 load balancers vs L7 proxies (a preview of Part 15). An L4 load balancer (AWS NLB, IPVS, Maglev) routes on the 5-tuple and passes the TCP stream through without reading it. An L7 proxy (ALB, nginx, Envoy) ends your TCP connection, reads the HTTP, and starts a new connection to the backend.