Chapter 22 — TCP part 2: how it never loses your data

Practice (30 min)

A. See sequence numbers and ACKs in a real transfer:

sudo tcpdump -i any -n -c 40 'tcp port 443 and host example.com'
# in another terminal:
curl -s https://example.com > /dev/null

Follow the seq and ack numbers down the capture. Watch seq climb by the length of each data packet, and ack climb to match.

B. Confirm SACK was negotiated — look at the SYN packets from Chapter 21 for sackOK. Then check your system's setting:

sysctl net.ipv4.tcp_sack        # Linux; should be 1

C. Cause real retransmissions and watch them. (Linux only; this deliberately degrades your network for a moment — do it on a test machine or be ready to undo.)

# Add 10% packet loss to outgoing traffic
sudo tc qdisc add dev eth0 root netem loss 10%

# Now download something and capture
sudo tcpdump -i eth0 -n 'host example.com' -w /tmp/loss.pcap &
curl -s https://example.com > /dev/null
sudo pkill tcpdump

# IMPORTANT: remove the impairment
sudo tc qdisc del dev eth0 root

Open /tmp/loss.pcap in Wireshark. It will highlight retransmissions and duplicate ACKs in black/red. You have just watched TCP heal itself.

D. Check your machine's retransmission counters:

# Linux
nstat -az | grep -Ei 'retrans|TCPLostRetransmit|TCPTimeouts'
# macOS
netstat -s -p tcp | grep -i retrans

A small number of retransmissions is normal. A rising percentage of segments retransmitted (over ~1%) indicates real trouble.

Build it in Go (25 min) — TCP is a byte stream, not a message stream

Everything in this chapter (sequence numbers, ACKs, retransmissions) works on bytes, not on your messages. TCP doesn't know or care where one of your Write calls ended and the next began. This surprises nearly every developer the first time they write a protocol by hand:

// stream: prove that TCP is a byte stream with no message boundaries, then fix
// it with length-prefix framing.
//
//	go run ./stream
package main

import (
	"bufio"
	"encoding/binary"
	"fmt"
	"io"
	"net"
	"time"
)

var messages = []string{"hello", "how are you?", "bye"}

func main() {
	ln, err := net.Listen("tcp", "127.0.0.1:0") // port 0: let the kernel pick a free port
	if err != nil {
		panic(err)
	}
	defer ln.Close()

	fmt.Println("== naive: one Write per message, one Read per message?")
	go naiveSender(ln.Addr().String())
	naiveReceiver(ln)

	fmt.Println("\n== framed: 4-byte length prefix before every message")
	go framedSender(ln.Addr().String())
	framedReceiver(ln)
}

func naiveSender(addr string) {
	c, _ := net.Dial("tcp", addr)
	defer c.Close()
	for _, m := range messages {
		c.Write([]byte(m)) // three separate Write calls...
	}
}

func naiveReceiver(ln net.Listener) {
	c, _ := ln.Accept()
	defer c.Close()
	time.Sleep(50 * time.Millisecond) // let all three writes land in the receive buffer
	buf := make([]byte, 1024)
	for {
		n, err := c.Read(buf)
		if err != nil {
			return
		}
		fmt.Printf("Read returned %2d bytes: %q\n", n, buf[:n]) // ...usually ONE Read
	}
}

func framedSender(addr string) {
	c, _ := net.Dial("tcp", addr)
	defer c.Close()
	w := bufio.NewWriter(c)
	for _, m := range messages {
		binary.Write(w, binary.BigEndian, uint32(len(m)))
		w.WriteString(m)
	}
	w.Flush() // one syscall for everything: framing makes batching safe
}

func framedReceiver(ln net.Listener) {
	c, _ := ln.Accept()
	defer c.Close()
	r := bufio.NewReader(c)
	for {
		var size uint32
		if err := binary.Read(r, binary.BigEndian, &size); err != nil {
			return // io.EOF: the sender closed cleanly
		}
		if size > 1<<20 {
			fmt.Println("refusing absurd frame size", size) // never trust a length from the wire
			return
		}
		msg := make([]byte, size)
		if _, err := io.ReadFull(r, msg); err != nil { // ReadFull loops over short reads
			fmt.Println("truncated frame:", err)
			return
		}
		fmt.Printf("message: %q\n", msg)
	}
}

$ go run ./stream
== naive: one Write per message, one Read per message?
Read returned 20 bytes: "hellohow are you?bye"

== framed: 4-byte length prefix before every message
message: "hello"
message: "how are you?"
message: "bye"

What to notice:

• Three writes arrived as one read. The bytes were all in the receive buffer by the time the reader looked, so one Read returned all of them. On a slow or lossy network the opposite also happens: one write can arrive across several reads. Both are correct TCP behaviour.
• io.ReadFull keeps reading until the buffer is full. Read alone may legally return fewer bytes, and assuming it won't is a classic production bug that only shows up under load or on bad networks.
• Validate lengths that come from the wire. A peer that sends a length of 4 GB would otherwise make you allocate 4 GB. The 1<<20 check is a one-line denial-of-service fix.
• Real protocols choose either delimiters (HTTP/1.1 headers end with \r\n\r\n; Redis uses \r\n) or length prefixes (HTTP/2 frames, gRPC messages, TLS records, Postgres, Kafka). Chapter 39 shows what happens when two parsers disagree about framing.

Exercises:

1. Remove the time.Sleep in naiveReceiver and run it 20 times. Does the split change from run to run? (It depends on timing, which is exactly the problem.)
2. Switch the framed version to newline delimiters with bufio.Scanner. Then send a message that contains a newline and watch it break. That's why binary protocols use length prefixes.
3. The Go guide's framing section and the Go plan's binary KV protocol (Day 53) take this to a full protocol with versioning and message types. Build the KV server.