Chapter 3 — TCP: Reliable Delivery

We have the IP address. Now we need to open a connection. TCP is the protocol that makes sure every byte arrives, in order, reliably.

TCP vs UDP — When to Use Which

Most HTTPS page loads historically used TCP with HTTP/1.1 or HTTP/2. Many large sites now also support HTTP/3 over QUIC, which uses UDP underneath but adds encryption, congestion control, stream management, and reliability at the QUIC layer.

The TCP 3-Way Handshake

Before any data is exchanged, TCP establishes a connection. This takes 1 Round Trip Time (RTT).

Client (your browser)              Server (Netflix edge, port 443)
        │                                      │
        │──── SYN ────────────────────────────►│
        │  seq=1000, SYN flag set               │
        │  "I want to connect. My starting      │
        │   sequence number is 1000."           │
        │                                      │
        │◄─── SYN-ACK ─────────────────────────│
        │  seq=5000, ack=1001                   │
        │  "OK. My starting seq is 5000.        │
        │   I confirm I received up to byte 1000│
        │   (ack=1001 means 'send me 1001 next')│
        │                                      │
        │──── ACK ────────────────────────────►│
        │  ack=5001                             │
        │  "Got it. Connection is open."        │
        │                                      │
        │         [CONNECTION ESTABLISHED]      │
        │  Data exchange can now begin          │

SYN = Synchronize (start a new connection) ACK = Acknowledge (confirm receipt)

The client and server each pick a random initial sequence number. This prevents attackers from guessing sequence numbers and injecting fake data.

TCP Segment Structure

Every chunk of data sent over TCP is wrapped in a segment with a header:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌─────────────────────────┬─────────────────────────────────────────┐
│      Source Port        │         Destination Port                │
├─────────────────────────┴─────────────────────────────────────────┤
│                        Sequence Number                            │
├───────────────────────────────────────────────────────────────────┤
│                     Acknowledgment Number                         │
├──────┬──────────┬─────────────────────────────────────────────────┤
│Offset│ Reserved │ Flags: URG ACK PSH RST SYN FIN                 │
├──────┴──────────┴─────────────────────────────────────────────────┤
│                Window Size              │       Checksum          │
├─────────────────────────────────────────┴─────────────────────────┤
│              Urgent Pointer             │       Options           │
├─────────────────────────────────────────────────────────────────── ┤
│                           DATA (payload)                          │
└───────────────────────────────────────────────────────────────────┘

Key fields:

• Source Port / Dest Port: Identifies the application (e.g., your browser tab on port 54321, Netflix on port 443).
• Sequence Number: Byte offset of this segment in the stream. Receiver uses this to reassemble data in order.
• Acknowledgment Number: "I have received all bytes up to this number. Send me this number next."
• Flags: Control bits — SYN, ACK, FIN, RST, PSH, URG.
• Window Size: How many bytes sender can transmit before needing an ACK (flow control).
• Checksum: Detects bit errors.

MSS — Maximum Segment Size

The maximum amount of data in a single TCP segment. Negotiated during the handshake.

Ethernet MTU: 1500 bytes
  - IP header:  20 bytes
  - TCP header: 20 bytes
  = MSS: 1460 bytes of actual data per segment

If a request is larger than 1460 bytes, TCP splits it into multiple segments automatically.

Reliable Delivery — How TCP Guarantees Nothing is Lost

Every segment must be acknowledged. If an ACK isn't received within a timeout period, the segment is retransmitted.

Client                          Server
  │                                │
  │──── Segment 1 (bytes 1-1460) ─►│
  │──── Segment 2 (bytes 1461-2920)►│
  │──── Segment 3 (bytes 2921-4380)►│ ← LOST in transit
  │                                │
  │◄─── ACK 2921 ──────────────────│  "I got up to byte 2920, send 2921 next"
  │                                │
  │  [Timeout waiting for ACK 4381]│
  │                                │
  │──── Segment 3 (retransmit) ───►│  Client resends the lost segment
  │                                │
  │◄─── ACK 4381 ──────────────────│  "Got it. Send 4381 next."

Cumulative ACKs: A single ACK covers all bytes received up to that point. If segments arrive out of order, the receiver holds them in a buffer and sends a Duplicate ACK (same ACK number repeated) to signal the gap.

Three Duplicate ACKs trigger Fast Retransmit — the sender immediately resends the missing segment without waiting for the timeout. Faster recovery than waiting for the timeout.

Flow Control — Window Size

The receiver tells the sender how much buffer space it has available via the Window Size field. The sender must not send more data than the receiver can buffer.

Server has 65535 bytes of receive buffer.
Server sends: Window=65535 "You can send up to 65535 bytes without waiting for ACK"

Client sends 65535 bytes.
Server processes some data, buffer empties.
Server sends: Window=32768 "I've made room. You can send 32768 more bytes."

If server sends Window=0:
  Client must stop sending. Waits for server to open the window.
  (This is called "zero window" — a common performance problem)

Congestion Control — Sharing the Network Fairly

The network between you and Netflix is shared by millions of users. If everyone sends at full speed, routers get overwhelmed and drop packets. TCP includes congestion control to prevent this.

Slow Start

When a connection opens, TCP starts conservatively and ramps up:

Round 1: Send 1 segment (1 × MSS)
Round 2: Send 2 segments (2 × MSS)   (doubles every RTT — exponential growth)
Round 3: Send 4 segments
Round 4: Send 8 segments
...
Until: Slow Start Threshold (ssthresh) is reached → switch to Congestion Avoidance

Congestion Avoidance: increase by 1 MSS per RTT (linear, careful growth)

If packet loss detected:
  → Assume network is congested
  → Cut window in half (or back to 1 with older algorithms)
  → Start over or resume from ssthresh

CUBIC (default on Linux)

CUBIC is the default congestion control algorithm on Linux servers (Netflix runs Linux). Instead of halving on loss, CUBIC uses a cubic function to probe for more bandwidth more aggressively after a loss, converging faster on the available bandwidth.

BBR (Bottleneck Bandwidth and Round-trip propagation time)

Google's newer algorithm. Instead of using packet loss as a congestion signal, BBR directly estimates the bottleneck bandwidth and RTT. Performs much better on long-latency paths and networks where some packet loss is normal (not caused by congestion). Netflix and Google use BBR for streaming.

Connection Teardown — Saying Goodbye

TCP closes gracefully with a 4-way FIN exchange:

Client                          Server
  │                                │
  │──── FIN ───────────────────────►│  "I'm done sending."
  │◄─── ACK ───────────────────────│  "OK, I received your FIN."
  │                                │  Server may still have data to send
  │◄─── FIN ───────────────────────│  "I'm done sending too."
  │──── ACK ───────────────────────►│  "Acknowledged."
  │                                │
        [CONNECTION CLOSED]

RST (Reset): For abrupt termination — "this connection is invalid, stop immediately." Used when a connection is refused or something goes wrong.

TIME_WAIT state: After sending the final ACK, the client waits 2×MSL (Maximum Segment Lifetime, typically 2 minutes) before fully closing. This ensures the final ACK is not lost and allows delayed packets to be discarded. This is why restarting servers sometimes fails to bind to the same port immediately — old sockets are in TIME_WAIT.

TCP Fast Open (TFO)

Normal TCP requires a full handshake (1 RTT) before data can be sent. TFO lets you send data along with the SYN packet on subsequent connections, saving 1 RTT.

First connection (normal):
  SYN → SYN-ACK → ACK → [data]     (1 RTT before data)

Server issues a TFO cookie (cryptographic token for your IP).

Subsequent connection (TFO):
  SYN + [data] + TFO cookie → SYN-ACK + [response data]
  (Data is in flight immediately with the SYN — 0 RTT for data)

TFO is supported by most modern OS and servers but disabled on some because 0-RTT data is replay-vulnerable (see TLS 0-RTT section).

Ports — Which App Gets the Packet?

Multiple applications run on the same server simultaneously. TCP port numbers (0–65535) route packets to the right application:

Packet arrives at Netflix server 10.0.4.55:
  Destination Port: 443  →  HTTPS server (Nginx)
  Destination Port: 22   →  SSH server
  Destination Port: 9200 →  Elasticsearch

Well-known ports:
  80   → HTTP
  443  → HTTPS
  22   → SSH
  53   → DNS
  25   → SMTP (email)
  3306 → MySQL
  6379 → Redis

A connection is uniquely identified by the 4-tuple: (src IP, src port, dst IP, dst port).