Step 1 — Check what you have

Run each of these. You want a version number, not "command not found".

ping -V 2>/dev/null || ping -c1 127.0.0.1   # reachability test
traceroute --version 2>/dev/null || traceroute -V
dig -v                                       # DNS lookups
curl --version                               # HTTP client
ss -V 2>/dev/null || netstat --version       # socket list (Linux: ss)
tcpdump --version                            # packet capture
nc -h 2>&1 | head -2                         # netcat: manual connections
ip -V 2>/dev/null || ifconfig -h 2>&1|head -1

Step 2 — Install what's missing

macOS (most tools ship built in):

# Homebrew: https://brew.sh  (install it first if you don't have it)
brew install iproute2mac mtr bind    # 'ip' command, mtr, and dig/host
brew install --cask wireshark        # GUI packet analyser

ping, traceroute, netstat, tcpdump, nc, curl, dig are already there. macOS has netstat rather than ss.

Linux (Debian/Ubuntu):

sudo apt update
sudo apt install -y iproute2 iputils-ping traceroute mtr-tiny dnsutils \
                    curl netcat-openbsd tcpdump wireshark whois

Windows: the friendliest path is WSL2 (Windows Subsystem for Linux) — install Ubuntu from the Microsoft Store, then use the Linux commands above. Native Windows equivalents also exist: ping, tracert, nslookup, netstat -ano, curl, and Wireshark.

Step 3 — Let yourself capture packets

Packet capture needs privileges.

Linux — either use sudo tcpdump ..., or grant your user permission once:

sudo usermod -aG wireshark $USER      # then LOG OUT and back in

macOS — sudo tcpdump ... works. Wireshark will prompt to install ChmodBPF.

Test it:

sudo tcpdump -i any -c 5 -n

You should see five lines of traffic scroll past. If you see tcpdump: <interface>: You don't have permission to capture, fix privileges above.

Step 4 — Know your own network

Run this and save the output in notes.md — you'll refer to it constantly:

# Linux
ip -br addr ; echo ---- ; ip route ; echo ---- ; cat /etc/resolv.conf

# macOS
ifconfig | grep -A3 '^en' ; echo ---- ; netstat -rn | head -20
echo ---- ; scutil --dns | grep nameserver | head

You are looking for four things:

1. Your IP address (something like 192.168.1.50)
2. Your subnet mask / prefix (/24 or 255.255.255.0)
3. Your default gateway (usually 192.168.1.1)
4. Your DNS server(s)

Don't worry about what they mean yet — Parts 3 and 4 explain each one. But having your own numbers in front of you makes every example concrete.

Practice (10 min)

Identify the layer for each of these, then check against Appendix F:

1. Your laptop's Wi-Fi card
2. 192.168.1.50
3. Your browser asking for /index.html
4. Port 443
5. a4:83:e7:2b:19:0c
6. Retransmitting a lost chunk of a file
7. example.com
8. A router deciding which cable to send a packet down

Then run this and label which layer each piece of output belongs to:

ip -br addr        # or on macOS: ifconfig
ip route           # or on macOS: netstat -rn
ss -tan | head     # or on macOS: netstat -an | head
curl -sI https://example.com | head -3

Practice (10 min) — measure it on your own machine

curl -w '
  dns lookup:   %{time_namelookup}s
  tcp connect:  %{time_connect}s
  tls setup:    %{time_appconnect}s
  first byte:   %{time_starttransfer}s
  total:        %{time_total}s
' -o /dev/null -s https://example.com

Each number is cumulative from the start. So:

• DNS took time_namelookup
• TCP handshake took time_connect − time_namelookup
• TLS took time_appconnect − time_connect
• The server's thinking + first byte took time_starttransfer − time_appconnect

Now run it again immediately. DNS should drop to nearly zero (it's cached). Run it against a server far away and compare (try https://www.gov.au or https://www.japan.go.jp). You have just measured the physics of the internet.

Practice (20 min)

A. See a real frame with its layers labelled:

sudo tcpdump -i any -c 3 -e -n

The -e flag shows the Ethernet header. You'll see aa:bb:cc:dd:ee:ff > 11:22:33:44:55:66, ethertype IPv4 (0x0800) at the front of each line. That's a real frame header.

B. Watch the MAC change but the IP stay the same. This is the key insight of Chapter 2, made visible:

# capture traffic to a website
sudo tcpdump -i any -c 5 -e -n 'host example.com'

Every packet you send to example.com has:

• Destination MAC = your router's MAC (the next hop), always
• Destination IP = 93.184.216.34 (the final destination), always

The MAC says "next door"; the IP says "eventually". Write that in notes.md.

C. Check for hardware errors (useful troubleshooting skill):

# Linux
ip -s link show eth0        # look for RX/TX 'errors' and 'dropped'
ethtool -S eth0 2>/dev/null | grep -Ei 'err|drop|crc'

# macOS
netstat -i                   # Ierrs / Oerrs columns

Non-zero and climbing CRC errors means a physical problem: a bad cable, a loose connector, interference. This is often the real cause of "the network is slow" and almost nobody checks it.

Practice (25 min)

1. Run the live capture above. Save the output in notes.md.
2. ip neigh (or arp -a) — how many neighbours does your machine know? Can you identify them (router, printer, phone, TV)?
3. Ping a device on your network that's turned off. Watch tcpdump: you'll see ARP requests repeated with no reply, and ping reports "Destination Host Unreachable". That error means ARP failed — the machine isn't there. This is a genuinely useful diagnostic to recognise.
4. Ping a device on the internet and watch ARP. You'll see... nothing new, because your router's MAC is already cached. Flush the cache and try again.

Practice (20 min)

A. See a software switch. Docker creates one on your machine:

# Linux, if you have Docker
bridge link                 # show bridge ports
bridge fdb show | head -20  # the MAC address table -- exactly as described above

B. Prove that a switch filters traffic. Ping between two devices on your network while capturing on a third device. With a switch, you should see almost none of their traffic (only broadcasts). With an old hub, you'd see everything.

C. Watch broadcasts:

sudo tcpdump -i any -n 'broadcast or multicast'

Leave it running for a minute. You'll see ARP requests, DHCP, mDNS (.local name discovery), SSDP (device discovery), and more. Every device on your network is doing this constantly. That's the broadcast domain at work.

Practice (20 min)

A. Look at your actual Wi-Fi quality:

# Linux
iw dev wlan0 link                # signal strength and current bitrate
iw dev wlan0 station dump | grep -E 'signal|bitrate|retries|failed'

# macOS
system_profiler SPAirPortDataType | grep -A10 "Current Network"
# or: wdutil info    (may need sudo)

Two numbers matter most:

• Signal: better than −60 dBm is good; worse than −75 dBm is trouble. (It's negative — closer to zero is stronger.)
• Tx bitrate: if it's far below your Wi-Fi standard's maximum, you're far from the access point or there's interference.

B. Watch it degrade. Run this while walking away from your router:

# Linux
watch -n1 'iw dev wlan0 link | grep -E "signal|bitrate"'

Watch the bitrate drop in steps as you move. That's the radio adapting, and it explains why your video call gets worse in the far room.

C. Compare latency stability. Ping your router over Wi-Fi and (if you can) over a cable:

ping -c 50 192.168.1.1

Look at the summary line: min/avg/max/mdev. mdev is the jitter. On a cable it'll be a fraction of a millisecond. On Wi-Fi it may be several milliseconds and occasionally much worse. That variability is why Wi-Fi feels different even when "fast".

Practice (15 min)

A. Find and understand your own address:

# Linux
ip -4 addr show
# macOS
ipconfig getifaddr en0 ; ifconfig en0 | grep 'inet '

Write down your address and note which of the private ranges it falls in.

B. See the loopback do nothing:

ping -c 2 127.0.0.1
sudo tcpdump -i any -c 5 -n 'host 127.0.0.1'   # in another terminal, then re-ping

Traffic to 127.0.0.1 appears on the lo interface and never touches your network card.

C. Convert to binary yourself (do at least two by hand before checking):

python3 -c "print('.'.join(format(int(o),'08b') for o in '192.168.1.50'.split('.')))"

Use the place-value table from Chapter 0.5: 192 = 128 + 64 so 11000000.

Practice (40 min) — do these on paper first

Set A — read the mask. For each, give network, broadcast, first/last usable, and host count. Answers in Appendix C.

   1.  192.168.10.75  /24
   2.  192.168.10.75  /26
   3.  10.5.3.200     /25
   4.  172.20.15.100  /28
   5.  10.0.0.6       /30
   6.  192.168.4.170  /27
   7.  172.16.35.42   /23
   8.  10.10.10.10    /22

Set B — are these two on the same network? Answers in Appendix C.

   9.  192.168.1.10/24   and  192.168.1.200/24
   10. 192.168.1.10/26   and  192.168.1.200/26
   11. 10.0.0.100/25     and  10.0.0.130/25
   12. 172.16.4.5/22     and  172.16.7.250/22

Set C — check your answers with a tool (only after doing them by hand):

# Install if needed: apt install ipcalc  /  brew install ipcalc
ipcalc 192.168.10.75/26

# Or use Python, which is everywhere:
python3 - <<'EOF'
import ipaddress
for cidr in ["192.168.10.75/24", "192.168.10.75/26", "10.5.3.200/25",
             "172.20.15.100/28", "10.0.0.6/30", "192.168.4.170/27",
             "172.16.35.42/23", "10.10.10.10/22"]:
    n = ipaddress.ip_network(cidr, strict=False)
    hosts = list(n.hosts())
    print(f"{cidr:20} network={n.network_address}  broadcast={n.broadcast_address}"
          f"  usable={hosts[0]}-{hosts[-1]}  count={len(hosts)}")
EOF

Set D — check your own network:

ip -4 addr show          # find your address and prefix, e.g. 192.168.1.50/24
ipcalc <your address>/<your prefix>

How many devices can your home network hold? Which addresses are usable?

Do not move on until you can do Set A without looking. Everything in Chapters 12–13 assumes it, and every networking job interview asks it.

Build it in Go (40 min) — a subnet calculator you can trust

Do the paper exercises above first. This program will check your answers, and writing it teaches you what the "four numbers" really are: bit operations on a 32-bit (or 128-bit) number.

Go's net/netip package represents addresses as small, comparable values (netip.Addr, netip.Prefix). It handles both IPv4 and IPv6 and gives you Masked(), Contains(), and Overlaps() for free. The program adds the parts that netip leaves to you: the broadcast address, the usable host range, and splitting a block into smaller ones.

Save as subnet/main.go:

// subnet: a subnet calculator and tiny IPAM tool built on net/netip.
//
//	go run ./subnet info  192.168.10.77/26
//	go run ./subnet split 10.0.0.0/16 24          # carve /24s out of a /16
//	go run ./subnet overlap 10.0.0.0/16 10.0.128.0/20 172.16.0.0/12
//	go run ./subnet alloc 10.0.0.0/16 22 10.0.0.0/24 10.0.4.0/22
package main

import (
	"fmt"
	"math/big"
	"net/netip"
	"os"
	"strconv"
)

func main() {
	if len(os.Args) < 3 {
		fmt.Println("usage: subnet info|split|overlap|alloc <args>")
		os.Exit(2)
	}
	var err error
	switch os.Args[1] {
	case "info":
		err = info(os.Args[2])
	case "split":
		err = split(os.Args[2], os.Args[3])
	case "overlap":
		err = overlap(os.Args[2:])
	case "alloc":
		err = alloc(os.Args[2], os.Args[3], os.Args[4:])
	default:
		err = fmt.Errorf("unknown command %q", os.Args[1])
	}
	if err != nil {
		fmt.Fprintln(os.Stderr, "error:", err)
		os.Exit(1)
	}
}

// info prints what Chapter 10 teaches you to work out by hand.
func info(s string) error {
	p, err := netip.ParsePrefix(s)
	if err != nil {
		return err
	}
	host := p.Addr() // the address as typed, e.g. 192.168.10.77
	p = p.Masked()   // zero the host bits -> the network address
	first, last := p.Addr(), lastAddr(p)
	fmt.Printf("input        %s\n", host)
	fmt.Printf("network      %s\n", p)
	if host.Is4() {
		fmt.Printf("netmask      %s\n", mask4(p.Bits()))
		fmt.Printf("broadcast    %s\n", last)
	}
	total := new(big.Int).Lsh(big.NewInt(1), uint(host.BitLen()-p.Bits()))
	usable := new(big.Int).Set(total)
	if host.Is4() && p.Bits() <= 30 { // network + broadcast are not usable hosts
		usable.Sub(usable, big.NewInt(2))
		first, last = first.Next(), last.Prev()
	}
	fmt.Printf("host range   %s - %s\n", first, last)
	fmt.Printf("addresses    %s total, %s usable\n", total, usable)
	fmt.Printf("private?     %v\n", host.IsPrivate())
	return nil
}

// split carves a prefix into equal smaller prefixes (Chapter 11's design exercise).
func split(s, bitsStr string) error {
	p, err := netip.ParsePrefix(s)
	if err != nil {
		return err
	}
	bits, err := strconv.Atoi(bitsStr)
	if err != nil || bits < p.Bits() || bits > p.Addr().BitLen() {
		return fmt.Errorf("new prefix length must be between /%d and /%d", p.Bits(), p.Addr().BitLen())
	}
	n := 0
	for sub := netip.PrefixFrom(p.Masked().Addr(), bits); p.Contains(sub.Addr()); {
		if n < 8 {
			fmt.Println(sub)
		}
		n++
		next := lastAddr(sub).Next()
		if !next.IsValid() { // ran off the end of the address space
			break
		}
		sub = netip.PrefixFrom(next, bits)
	}
	if n > 8 {
		fmt.Printf("... %d subnets in total\n", n)
	}
	return nil
}

// overlap reports every pair of prefixes that overlap -- the check that would
// have saved the "two companies both picked 10.0.0.0/16" migration.
func overlap(args []string) error {
	ps, err := parseAll(args)
	if err != nil {
		return err
	}
	clean := true
	for i := range ps {
		for j := i + 1; j < len(ps); j++ {
			if ps[i].Overlaps(ps[j]) {
				fmt.Printf("OVERLAP  %s  <->  %s\n", ps[i], ps[j])
				clean = false
			}
		}
	}
	if clean {
		fmt.Println("no overlaps")
	}
	return nil
}

// alloc finds the first free /bits block inside pool that overlaps none of
// the already-used prefixes. This is the core of every IPAM system.
func alloc(poolStr, bitsStr string, usedStr []string) error {
	pool, err := netip.ParsePrefix(poolStr)
	if err != nil {
		return err
	}
	bits, err := strconv.Atoi(bitsStr)
	if err != nil {
		return err
	}
	used, err := parseAll(usedStr)
	if err != nil {
		return err
	}
	for cand := netip.PrefixFrom(pool.Masked().Addr(), bits); pool.Contains(cand.Addr()); {
		free := true
		for _, u := range used {
			if cand.Overlaps(u) {
				free = false
				break
			}
		}
		if free {
			fmt.Println("allocated", cand)
			return nil
		}
		next := lastAddr(cand).Next()
		if !next.IsValid() {
			break
		}
		cand = netip.PrefixFrom(next, bits)
	}
	return fmt.Errorf("pool %s has no free /%d", pool, bits)
}

func parseAll(ss []string) ([]netip.Prefix, error) {
	var out []netip.Prefix
	for _, s := range ss {
		p, err := netip.ParsePrefix(s)
		if err != nil {
			return nil, err
		}
		out = append(out, p.Masked())
	}
	return out, nil
}

// lastAddr sets every host bit to 1: the broadcast address for IPv4.
func lastAddr(p netip.Prefix) netip.Addr {
	b := p.Masked().Addr().AsSlice()
	for i := p.Bits(); i < len(b)*8; i++ {
		b[i/8] |= 1 << (7 - uint(i%8))
	}
	a, _ := netip.AddrFromSlice(b)
	return a
}

func mask4(bits int) netip.Addr {
	var b [4]byte
	for i := 0; i < bits; i++ {
		b[i/8] |= 1 << (7 - uint(i%8))
	}
	return netip.AddrFrom4(b)
}

Run it against the worked examples:

$ go run ./subnet info 192.168.10.77/26
input        192.168.10.77
network      192.168.10.64/26
netmask      255.255.255.192
broadcast    192.168.10.127
host range   192.168.10.65 - 192.168.10.126
addresses    64 total, 62 usable
private?     true

$ go run ./subnet split 10.0.0.0/16 24
10.0.0.0/24
10.0.1.0/24
...
... 256 subnets in total

What to notice in the code:

• p.Masked() is "AND the address with the mask". lastAddr is "set every host bit to 1". The block-size shortcut from this chapter is the same bit arithmetic done in your head.
• The usable count subtracts 2 only for IPv4 prefixes of /30 or larger. A /31 (point-to-point links, RFC 3021) and a /32 (a single host) don't sacrifice a network/broadcast address, and IPv6 has no broadcast at all.
• math/big is needed because a single IPv6 /64 holds 2⁶⁴ addresses, which doesn't fit in a signed 64-bit integer.

Exercises:

1. Run info on all of Set A and Set B from the practice above, and compare with your paper answers.
2. Add a contains command: subnet contains 192.168.1.0/25 192.168.1.200 prints yes/no. This is the "can I reach it directly?" test from this chapter, the same decision your OS makes before choosing ARP or the gateway.
3. Add -json output. You'll reuse it in Chapter 61's IPAM exercise.

Step 1 — size each subnet

Step 2 & 3 — allocate, largest first

   Start at 192.168.10.0

   1. STAFF      /25   block 128   ->  192.168.10.0   -  192.168.10.127
                                       network .0     broadcast .127
                                       usable  .1  to .126        (126 addresses)

   2. GUESTS     /26   block  64   ->  192.168.10.128 -  192.168.10.191
                                       usable  .129 to .190       (62)

   3. SERVERS    /28   block  16   ->  192.168.10.192 -  192.168.10.207
                                       usable  .193 to .206       (14)

   4. PRINTERS   /28   block  16   ->  192.168.10.208 -  192.168.10.223
                                       usable  .209 to .222       (14)

   5. ROUTER LINK /30  block   4   ->  192.168.10.224 -  192.168.10.227
                                       usable  .225 to .226       (2)

   LEFT OVER: 192.168.10.228 - 192.168.10.255  (28 addresses for growth)

Practice (45 min)

Problem 1. You have 10.20.0.0/24. Allocate for:

• Sales: 60 devices
• Engineering: 25 devices
• Management: 10 devices
• Wi-Fi guests: 100 devices
• Two point-to-point router links: 2 devices each

Give the network address, mask, usable range, and broadcast for each.

Problem 2. You have 172.16.0.0/22 (1024 addresses). Allocate for:

• Floor 1: 200 devices
• Floor 2: 200 devices
• Floor 3: 100 devices
• Servers: 50 devices
• Management VLAN: 20 devices

Problem 3 — reverse engineering. Given these three subnets, what is the smallest single block that contains all of them?

   192.168.4.0/24
   192.168.5.0/24
   192.168.6.0/24
   192.168.7.0/24

(Answers to all three: Appendix C.)

Verify with a tool:

python3 - <<'EOF'
import ipaddress
block = ipaddress.ip_network("192.168.10.0/24")
# subnet_of / supernet helpers are handy for checking your work
for pfx in [25, 26, 28, 28, 30]:
    sub = next(block.subnets(new_prefix=pfx))
    print(pfx, sub, list(sub.hosts())[0], list(sub.hosts())[-1], sub.broadcast_address)
    # then manually advance -- this is the exercise
EOF

Build it in Go (30 min) — check your design with code

Use the subnet tool from Chapter 10 to check the design you just did by hand:

$ go run ./subnet overlap 10.0.0.0/25 10.0.0.128/26 10.0.0.192/27 10.0.0.224/30
no overlaps

$ go run ./subnet overlap 10.0.0.0/16 10.0.128.0/20 172.16.0.0/12
OVERLAP  10.0.0.0/16  <->  10.0.128.0/20

$ go run ./subnet alloc 10.0.0.0/16 22 10.0.0.0/24 10.0.4.0/22
allocated 10.0.8.0/22

alloc is "largest first" turned into code: walk the pool in steps of the requested size and return the first block that overlaps nothing already used. Because it only ever proposes aligned blocks (/22s start on multiples of 4 in the third octet), it can't produce an invalid subnet. That is the mistake people make most often by hand.

Exercise: write plan that takes a pool and a list of host counts (subnet plan 10.0.0.0/24 100 50 20 2), sorts them largest first, works out the prefix length each needs (hosts + 2, rounded up to a power of two), and allocates them in order with alloc. You have just automated this chapter. Chapter 61 grows it into an IPAM.

Practice (20 min)

A. See both of your addresses:

# your PRIVATE address (what your device thinks it is)
ip -4 addr show | grep inet        # Linux
ipconfig getifaddr en0             # macOS

# your PUBLIC address (what the internet sees)
curl -s ifconfig.me ; echo

They're different. Everyone in your house shows the same public one. Try it on your phone (on Wi-Fi) to confirm.

B. Find your router's NAT table (if it's a Linux box or you can log into your home router's web UI — look for "NAT", "connections", or "active sessions"):

# On a Linux machine acting as a router:
sudo conntrack -L | head

C. Prove NAT blocks inbound. Start a simple server on your laptop:

python3 -m http.server 8000

• From another device in your house: http://<your-private-ip>:8000 → works.
• From outside (mobile data, or ask a friend): http://<your-public-ip>:8000 → fails, because no NAT mapping exists. That's the accidental firewall.

D. Check whether you're behind CGNAT: compare curl ifconfig.me with the WAN address shown in your router's admin page. If your router's WAN address is in 100.64.x.x–100.127.x.x, you are behind carrier-grade NAT.

Build it in Go (15 min) — see your own NAT

Two questions decide whether you're behind NAT: what address does my machine think it has? and what address does the internet see? This program asks both.

// whoami: show the NAT in front of you by comparing the address your machine
// THINKS it has with the address the internet SEES.
//
//	go run ./whoami
package main

import (
	"context"
	"fmt"
	"io"
	"net"
	"net/http"
	"strings"
	"time"
)

func main() {
	// 1. Which local address would the kernel use to reach the internet?
	//    "Dialing" UDP sends no packets -- it only asks the kernel to pick a
	//    route and a source address, which we can then read back.
	c, err := net.Dial("udp", "1.1.1.1:53")
	if err != nil {
		fmt.Println("no route to the internet:", err)
		return
	}
	local := c.LocalAddr().(*net.UDPAddr).IP
	c.Close()
	fmt.Println("private (what you think you are): ", local)

	// 2. Ask a server on the internet which address our packets arrived from.
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	req, _ := http.NewRequestWithContext(ctx, "GET", "https://api.ipify.org", nil)
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		fmt.Println("could not reach api.ipify.org:", err)
		return
	}
	defer resp.Body.Close()
	body, _ := io.ReadAll(io.LimitReader(resp.Body, 64))
	public := net.ParseIP(strings.TrimSpace(string(body)))
	fmt.Println("public  (what the internet sees): ", public)

	// 3. Interpret.
	switch {
	case public == nil:
		fmt.Println("unexpected answer:", string(body))
	case local.Equal(public):
		fmt.Println("=> no NAT: your machine holds a public address directly")
	case isCGNAT(local):
		fmt.Println("=> carrier-grade NAT: even your 'router' address is shared (100.64.0.0/10)")
	default:
		fmt.Println("=> NAT: a router rewrote your source address on the way out")
	}
}

func isCGNAT(ip net.IP) bool {
	_, cgnat, _ := net.ParseCIDR("100.64.0.0/10")
	return cgnat.Contains(ip)
}

$ go run ./whoami
private (what you think you are):  192.168.1.2
public  (what the internet sees):  122.171.18.134
=> NAT: a router rewrote your source address on the way out

What to notice: net.Dial("udp", ...) sends no packet. UDP has no handshake, so "connecting" a UDP socket only asks the kernel to pick a route and a source address. Reading LocalAddr() afterwards is a standard trick for finding "my outbound IP" without parsing routing tables.

Exercises:

1. Run it on home Wi-Fi, then on a phone hotspot. On mobile networks you'll often get the carrier-grade NAT result (100.64.0.0/10), or a public address that thousands of other customers share at the same moment.
2. Run it inside a Docker container (docker run --rm -v "$PWD":/w -w /w golang:1.26 go run ./whoami). You'll see an extra layer of NAT: container → host → router.
3. Why does this matter for software? Read the "rate limiting by IP" point in the OSI guide's Layer 3 chapter, then decide what your service should use as a client identity.

Practice (30 min)

A. Read your own routing table and explain each line out loud.

ip route          # or: netstat -rn

B. Ask your machine to decide (Linux — this is a great tool):

ip route get 8.8.8.8
ip route get 192.168.1.99      # substitute an address on your own network
ip route get 127.0.0.1

It prints exactly which route it chose, which interface, and which source address. Compare all three answers — one goes via the gateway, one goes direct, one stays local.

C. See the hops for real:

traceroute -n example.com          # macOS/Linux
tracert example.com                # Windows
mtr -rwzbc 50 example.com          # better: continuous, shows loss per hop

Each line is one router along the path. Chapter 14 explains exactly how this works and — importantly — how to read it without being misled.

D. Watch TTL decrease. Ping something far away and look at the TTL in the reply:

ping -c 1 example.com
# "64 bytes from ...: icmp_seq=0 ttl=52 time=91 ms"

Most systems start TTL at 64. A reply arriving with ttl=52 means it crossed about 64 − 52 = 12 routers on the way back.

Practice (30 min)

A. Ping a few destinations and compare:

ping -c 10 192.168.1.1              # your router: should be <5 ms, near-zero jitter
ping -c 10 1.1.1.1                  # a big public DNS: typically 5-30 ms
ping -c 10 example.com              # a specific server, maybe far away

Record the min/avg/max/mdev for each in notes.md. You are measuring distance. Roughly: light in fibre travels ~200 km per millisecond, and a round trip doubles it — so 90 ms round trip ≈ 9,000 km of fibre, plus routing overhead.

B. Traceroute somewhere far away and count the hops:

traceroute -n www.gov.au        # or any distant site

Can you spot where it leaves your country? (Hostnames often hint at cities — try without -n.)

C. Run mtr for two minutes to a site you use daily, and identify: which hop is your router, which is your ISP, where loss (if any) begins.

D. Watch ICMP in tcpdump:

sudo tcpdump -i any -n icmp
# then, in another terminal:  ping -c 3 1.1.1.1

You'll see the request and reply pairs.

E. Trigger an error deliberately:

# ask for a UDP port nothing is listening on
sudo tcpdump -i any -n icmp &
nc -u -w1 127.0.0.1 9   # or any closed UDP port

You should see an ICMP port unreachable.

Practice (25 min)

A. Find your own path MTU to three destinations: your router, a nearby site, a distant site. Record them.

B. Break it on purpose (safe, and instructive):

# Linux -- lower your MTU, see what happens, then put it back
ip link show eth0 | grep mtu           # note the current value
sudo ip link set eth0 mtu 1200
curl -s -o /dev/null -w '%{http_code}\n' https://example.com
sudo ip link set eth0 mtu 1500         # PUT IT BACK

Small MTU still works (just less efficiently) because PMTUD is functioning. The breakage comes from a mismatch plus blocked ICMP.

C. See fragmentation:

sudo tcpdump -i any -n 'ip[6:2] & 0x3fff != 0'   # captures only fragments
# in another terminal:
ping -s 4000 -c 1 192.168.1.1      # a 4000-byte ping must be fragmented

D. If you use a VPN, run the MTU test with the VPN on and off. Compare.

Practice (20 min)

A. Do you have IPv6?

ip -6 addr show                 # Linux
ifconfig | grep inet6           # macOS

curl -6 -s https://ifconfig.co ; echo    # your public IPv6 (fails if you have none)
curl -4 -s https://ifconfig.co ; echo    # your public IPv4

Also visit test-ipv6.com in a browser — it gives a clear score.

B. Note that you always have a link-local address, even with no IPv6 internet:

ip -6 addr show | grep fe80

Every IPv6-capable interface has one, always.

C. Ping over IPv6:

ping6 -c 3 2606:4700:4700::1111        # Cloudflare's IPv6 DNS
ping6 -c 3 ipv6.google.com

# link-local needs an interface specified, note the % syntax:
ping6 -c 2 fe80::1%eth0

D. Watch Neighbor Discovery (IPv6's ARP):

sudo tcpdump -i any -n 'icmp6 && ip6[40] >= 133 && ip6[40] <= 137'

Leave it running a minute. You'll see Router Advertisements and Neighbor Solicitations — the IPv6 equivalents of Chapter 6's ARP.

E. Compare DNS answers:

dig example.com A +short
dig example.com AAAA +short

Practice (25 min)

A. Watch DORA happen live. This is very satisfying.

# Terminal 1 -- start capturing
sudo tcpdump -i any -n -v 'udp port 67 or udp port 68'

# Terminal 2 -- force a renewal
# Linux (NetworkManager):
sudo nmcli device reapply <your-interface>
#   or, more forcefully:
sudo dhclient -r && sudo dhclient
# macOS:
sudo ipconfig set en0 DHCP

You'll see the DISCOVER / OFFER / REQUEST / ACK sequence, with all the options listed. Save a copy in notes.md.

B. Look at your current lease:

# Linux
nmcli -f DHCP4 connection show "<connection name>"
cat /var/lib/dhcp/dhclient.leases 2>/dev/null | tail -30

# macOS
ipconfig getpacket en0

Find: your address, lease time, gateway, DNS servers.

C. Log into your router's admin page (usually http://192.168.1.1) and find the DHCP settings. Look at:

• The address pool range
• The lease time
• The list of currently connected devices

Try setting a reservation for one device (bind its MAC to a fixed address) — this is how you give a printer or server a stable address without configuring it manually.

Practice (35 min) — this is the most useful tool chapter in Part 4

A. Basic lookups:

dig example.com                     # full output -- read the ANSWER section
dig example.com +short              # just the answer
dig example.com A +short
dig example.com AAAA +short
dig example.com MX +short
dig example.com NS +short
dig example.com TXT +short

B. Watch the whole delegation chain. This is the single best DNS learning command:

dig +trace example.com

You will literally see it start at the root servers, get referred to .com, then to example.com's own servers, then get the answer. Run this now. It makes the whole chapter concrete.

C. See caching in action:

dig example.com | grep -E "Query time|ANSWER SECTION" -A2
# run it again immediately
dig example.com | grep "Query time"

The second query time should be dramatically lower — and watch the TTL count down between runs.

D. Ask a specific server (bypasses your local cache):

dig @1.1.1.1 example.com +short          # Cloudflare
dig @8.8.8.8 example.com +short          # Google
dig @ns1.example.com example.com +short  # straight to the source (authoritative)

If different servers give different answers, you've found a propagation or split-horizon issue.

E. Reverse lookup:

dig -x 93.184.216.34 +short

F. Watch DNS on the wire:

sudo tcpdump -i any -n 'udp port 53'
# in another terminal:
dig example.com

You'll see the query go out and the response come back — usually one packet each.

G. See your own resolver settings:

cat /etc/resolv.conf              # Linux
scutil --dns | head -20           # macOS
resolvectl status                 # Linux with systemd-resolved

Build it in Go (60 min) — a DNS client from raw bytes

dig is a wrapper around a surprisingly small protocol. This program builds a DNS query byte by byte, sends it in one UDP datagram, and decodes the reply, including the "message compression" pointers that confuse everyone the first time. It uses no DNS library: only encoding/binary and a UDP socket.

The wire format you're implementing (RFC 1035):

 HEADER (12 bytes)   ID | flags (QR, RD, RA, TC, rcode) | QDCOUNT | ANCOUNT | NSCOUNT | ARCOUNT
 QUESTION            name as labels: 7 e x a m p l e 3 c o m 0 | QTYPE (A=1) | QCLASS (IN=1)
 ANSWER records      name (often a 2-byte POINTER: 0xC0 0x0C) | TYPE | CLASS | TTL | RDLENGTH | RDATA

Save as dnsquery/main.go:

// dnsquery: build a DNS query by hand, send it over UDP, and decode the answer.
// No DNS library -- just the bytes RFC 1035 describes.
//
//	go run ./dnsquery example.com
//	go run ./dnsquery -type AAAA -server 8.8.8.8 example.com
//	go run ./dnsquery -type CNAME www.github.com
package main

import (
	"encoding/binary"
	"errors"
	"flag"
	"fmt"
	"math/rand/v2"
	"net"
	"os"
	"strings"
	"time"
)

var qtypes = map[string]uint16{"A": 1, "NS": 2, "CNAME": 5, "MX": 15, "TXT": 16, "AAAA": 28}

func main() {
	server := flag.String("server", "1.1.1.1", "recursive resolver to ask")
	qtype := flag.String("type", "A", "record type: A, AAAA, CNAME, NS, MX, TXT")
	flag.Parse()
	if flag.NArg() != 1 {
		fmt.Println("usage: dnsquery [-server IP] [-type T] name")
		os.Exit(2)
	}
	t, ok := qtypes[strings.ToUpper(*qtype)]
	if !ok {
		fmt.Println("unsupported type", *qtype)
		os.Exit(2)
	}

	id := uint16(rand.N(1 << 16))
	query := buildQuery(id, flag.Arg(0), t)
	fmt.Printf("query: %d bytes  % x ...\n", len(query), query[:12])

	conn, err := net.Dial("udp", net.JoinHostPort(*server, "53"))
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	defer conn.Close()
	conn.SetDeadline(time.Now().Add(3 * time.Second)) // UDP has no retries: we own the timeout

	start := time.Now()
	if _, err := conn.Write(query); err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	buf := make([]byte, 1232) // the EDNS-era "safe" UDP DNS size
	n, err := conn.Read(buf)
	if err != nil {
		fmt.Println("no answer:", err)
		os.Exit(1)
	}
	fmt.Printf("reply: %d bytes from %s in %v\n\n", n, *server, time.Since(start).Round(time.Microsecond))
	if err := printReply(buf[:n], id); err != nil {
		fmt.Println("bad reply:", err)
		os.Exit(1)
	}
}

// buildQuery writes the 12-byte header plus one question.
func buildQuery(id uint16, name string, qtype uint16) []byte {
	b := make([]byte, 12, 512)
	binary.BigEndian.PutUint16(b[0:], id)
	binary.BigEndian.PutUint16(b[2:], 0x0100) // flags: RD=1 ("recursion desired")
	binary.BigEndian.PutUint16(b[4:], 1)      // QDCOUNT = 1 question
	// a name is a sequence of length-prefixed labels: 7example3com0
	for _, label := range strings.Split(strings.TrimSuffix(name, "."), ".") {
		b = append(b, byte(len(label)))
		b = append(b, label...)
	}
	b = append(b, 0)
	b = binary.BigEndian.AppendUint16(b, qtype)
	b = binary.BigEndian.AppendUint16(b, 1) // class IN (internet)
	return b
}

func printReply(msg []byte, wantID uint16) error {
	if len(msg) < 12 {
		return errors.New("short header")
	}
	if id := binary.BigEndian.Uint16(msg[0:]); id != wantID {
		return fmt.Errorf("ID mismatch: sent %d got %d (spoofed or stale reply?)", wantID, id)
	}
	flags := binary.BigEndian.Uint16(msg[2:])
	rcode := flags & 0xF
	qd, an := binary.BigEndian.Uint16(msg[4:]), binary.BigEndian.Uint16(msg[6:])
	fmt.Printf("flags: QR=%d AA=%d TC=%d RD=%d RA=%d  rcode=%s  answers=%d\n",
		flags>>15&1, flags>>10&1, flags>>9&1, flags>>8&1, flags>>7&1, rcodeName(rcode), an)
	if flags>>9&1 == 1 {
		fmt.Println("TC=1: answer truncated -- a real resolver now retries over TCP")
	}

	off := 12
	for i := 0; i < int(qd); i++ { // skip the echoed question
		_, next, err := readName(msg, off)
		if err != nil {
			return err
		}
		off = next + 4
	}
	for i := 0; i < int(an); i++ {
		name, next, err := readName(msg, off)
		if err != nil {
			return err
		}
		if next+10 > len(msg) {
			return errors.New("truncated record header")
		}
		typ := binary.BigEndian.Uint16(msg[next:])
		ttl := binary.BigEndian.Uint32(msg[next+4:])
		rdlen := int(binary.BigEndian.Uint16(msg[next+8:]))
		rdata := next + 10
		if rdata+rdlen > len(msg) {
			return errors.New("truncated rdata")
		}
		fmt.Printf("%-30s TTL=%-6d %-5s %s\n", name, ttl, typeName(typ), rdataString(msg, typ, rdata, rdlen))
		off = rdata + rdlen
	}
	return nil
}

// readName decodes a possibly-compressed name. A byte >= 0xC0 starts a
// 2-byte pointer to an earlier offset: that is DNS "message compression".
func readName(msg []byte, off int) (string, int, error) {
	var labels []string
	next := -1 // where parsing continues after the name (set at first pointer)
	for hops := 0; hops < 20; hops++ {
		if off >= len(msg) {
			return "", 0, errors.New("name runs past end")
		}
		l := int(msg[off])
		switch {
		case l == 0:
			if next < 0 {
				next = off + 1
			}
			return strings.Join(labels, ".") + ".", next, nil
		case l&0xC0 == 0xC0:
			if off+1 >= len(msg) {
				return "", 0, errors.New("bad pointer")
			}
			if next < 0 {
				next = off + 2
			}
			off = int(binary.BigEndian.Uint16(msg[off:]) & 0x3FFF)
		default:
			if off+1+l > len(msg) {
				return "", 0, errors.New("label runs past end")
			}
			labels = append(labels, string(msg[off+1:off+1+l]))
			off += 1 + l
		}
	}
	return "", 0, errors.New("compression loop")
}

func rdataString(msg []byte, typ uint16, off, n int) string {
	switch typ {
	case 1, 28: // A, AAAA
		return net.IP(msg[off : off+n]).String()
	case 2, 5: // NS, CNAME
		name, _, err := readName(msg, off)
		if err != nil {
			return "?"
		}
		return name
	case 15: // MX: preference + name
		name, _, _ := readName(msg, off+2)
		return fmt.Sprintf("%d %s", binary.BigEndian.Uint16(msg[off:]), name)
	case 16: // TXT: one or more length-prefixed strings
		var parts []string
		for i := off; i < off+n; {
			l := int(msg[i])
			if i+1+l > off+n {
				break
			}
			parts = append(parts, fmt.Sprintf("%q", msg[i+1:i+1+l]))
			i += 1 + l
		}
		return strings.Join(parts, " ")
	}
	return fmt.Sprintf("% x", msg[off:off+n])
}

func typeName(t uint16) string {
	for k, v := range qtypes {
		if v == t {
			return k
		}
	}
	return fmt.Sprint(t)
}

func rcodeName(r uint16) string {
	switch r {
	case 0:
		return "NOERROR"
	case 2:
		return "SERVFAIL"
	case 3:
		return "NXDOMAIN"
	case 5:
		return "REFUSED"
	}
	return fmt.Sprint(r)
}

Real output:

$ go run ./dnsquery example.com
query: 29 bytes  49 8d 01 00 00 01 00 00 00 00 00 00 ...
reply: 61 bytes from 1.1.1.1 in 17.618ms

flags: QR=1 AA=0 TC=0 RD=1 RA=1  rcode=NOERROR  answers=2
example.com.                   TTL=179    A     172.66.147.243
example.com.                   TTL=179    A     104.20.23.154

$ go run ./dnsquery www.github.com
www.github.com.                TTL=3040   CNAME github.com.
github.com.                    TTL=8      A     20.207.73.82

$ go run ./dnsquery nonexistent-zzzz.example
flags: QR=1 AA=0 TC=0 RD=1 RA=1  rcode=NXDOMAIN  answers=0

What to notice:

• The whole query is 29 bytes. DNS is fast because it's one small datagram each way, with no handshake.
• AA=0: the answer came from the resolver's cache, not from the authoritative server. Run it twice and watch the TTL count down. That's the cache ageing in real time.
• The ID check matters for security. A reply with the wrong ID is rejected. Off-path DNS spoofing attacks (Kaminsky, 2008) work by guessing this 16-bit ID, which is why resolvers also randomise the source port.
• The CNAME chain is in one reply. The resolver followed www.github.com → github.com for you and returned both records.
• TC=1 means the answer didn't fit in the UDP packet; real clients retry over TCP port 53. Our 1232-byte buffer is the size the DNS community agreed in 2020 avoids IP fragmentation on almost every path.

Exercises:

1. Point it at a server that doesn't exist (-server 192.0.2.1). You get a timeout, not an error, because UDP can't tell "no server" from "slow server". This is why a dead resolver in /etc/resolv.conf adds a fixed 5-second delay (the "suspiciously round number" from Chapter 26).
2. Query an authoritative server directly: find it with go run ./dnsquery -type NS example.com, resolve it, then ask it with -server. Check that AA=1.
3. Add retries with exponential backoff (1 s, 2 s, 4 s) and query two servers in parallel, using whichever answers first. That is what your OS stub resolver does.
4. Stretch: implement the iterative algorithm. Start at a root server (198.41.0.4), follow NS referrals in the authority section, and reproduce dig +trace yourself.

Across the series: the HTTPS guide's DNS chapter follows this lookup from the browser's point of view (browser cache, OS cache, DoH, HTTPS records). The Go standard library's resolver is net.Resolver; Chapter 55's dnsdiff lab uses it to query many servers at once.

Practice (25 min)

A. See what's listening on your machine:

# Linux
ss -tulnp

# macOS
sudo lsof -iTCP -sTCP:LISTEN -n -P
netstat -an | grep LISTEN

For each one, ask: do I know what that is, and should it be reachable from the network? This is a genuinely useful security habit.

B. See your active connections:

ss -tn state established        # Linux
netstat -an | grep ESTABLISHED  # macOS

Open a few browser tabs and run it again. Watch the count grow.

C. Be a server and a client yourself — this makes ports concrete:

# Terminal 1: listen on port 9999
nc -l 9999

# Terminal 2: connect to it and type something
nc 127.0.0.1 9999
# whatever you type appears in terminal 1. You just built a chat app.

# Terminal 3: watch the connection exist
ss -tn '( sport = :9999 or dport = :9999 )'

D. Prove the binding-address rule:

# Bind to loopback only
python3 -m http.server 8000 --bind 127.0.0.1
# From this machine:      curl http://127.0.0.1:8000   -> works
# From another device:    curl http://<your-ip>:8000    -> FAILS

# Now bind to everything
python3 -m http.server 8000 --bind 0.0.0.0
# From another device: works.

E. Check whether a remote port is open:

nc -zv example.com 443        # "succeeded" or "refused"/timeout
nc -zv example.com 444        # almost certainly nothing there

Build it in Go (25 min) — the binding rules, tried for real

Three rules from this chapter cause real outages: one listener per address:port, a loopback bind only accepts loopback traffic, and the less well-known SO_REUSEPORT, which lets several sockets share one port on purpose.

//go:build linux || darwin

// bindlab: Chapter 19's binding rules, tried for real.
//
//	go run ./bindlab conflict     # two listeners on one port -> EADDRINUSE
//	go run ./bindlab reuseport    # SO_REUSEPORT: four listeners share one port
//	go run ./bindlab loopback     # 127.0.0.1 vs 0.0.0.0, and what each accepts
package main

import (
	"context"
	"errors"
	"fmt"
	"net"
	"os"
	"sync"
	"syscall"
	"time"
)

func main() {
	if len(os.Args) != 2 {
		fmt.Println("usage: bindlab conflict|reuseport|loopback")
		os.Exit(2)
	}
	switch os.Args[1] {
	case "conflict":
		conflict()
	case "reuseport":
		reusePort()
	case "loopback":
		loopback()
	}
}

func conflict() {
	a, err := net.Listen("tcp", "127.0.0.1:7300")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer a.Close()
	_, err = net.Listen("tcp", "127.0.0.1:7300")
	fmt.Println("second listen:", err)
	fmt.Println("EADDRINUSE?   ", errors.Is(err, syscall.EADDRINUSE))
}

// reusePort opens four sockets on the SAME port with SO_REUSEPORT set before
// bind(). The kernel then spreads incoming connections across them -- the
// trick nginx, Envoy, and HAProxy use to scale accept() across CPU cores.
func reusePort() {
	lc := net.ListenConfig{
		Control: func(network, address string, c syscall.RawConn) error {
			var opErr error
			err := c.Control(func(fd uintptr) {
				opErr = syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET, syscall.SO_REUSEPORT, 1)
			})
			if err != nil {
				return err
			}
			return opErr
		},
	}
	counts := make([]int, 4)
	var mu sync.Mutex
	for i := range counts {
		ln, err := lc.Listen(context.Background(), "tcp", "127.0.0.1:7301")
		if err != nil {
			fmt.Println("listener", i, err)
			return
		}
		go func() {
			for {
				c, err := ln.Accept()
				if err != nil {
					return
				}
				mu.Lock()
				counts[i]++
				mu.Unlock()
				c.Close()
			}
		}()
	}
	for i := 0; i < 400; i++ {
		if c, err := net.Dial("tcp", "127.0.0.1:7301"); err == nil {
			c.Close()
		}
	}
	time.Sleep(200 * time.Millisecond)
	mu.Lock()
	defer mu.Unlock()
	fmt.Println("connections accepted per listener:", counts)
	fmt.Println("(Linux hashes each connection to one socket -> roughly even;")
	fmt.Println(" macOS/BSD semantics differ -> often all on one listener)")
}

func loopback() {
	lo, _ := net.Listen("tcp", "127.0.0.1:7302")
	all, _ := net.Listen("tcp", "0.0.0.0:7303")
	defer lo.Close()
	defer all.Close()

	// Find a non-loopback address of this machine to test "from the network".
	c, err := net.Dial("udp", "1.1.1.1:53")
	if err != nil {
		fmt.Println("no network address to test with:", err)
		return
	}
	lan := c.LocalAddr().(*net.UDPAddr).IP.String()
	c.Close()

	for _, t := range []string{"127.0.0.1:7302", lan + ":7302", "127.0.0.1:7303", lan + ":7303"} {
		conn, err := net.DialTimeout("tcp", t, time.Second)
		if err != nil {
			fmt.Printf("%-22s FAIL  %v\n", t, err)
			continue
		}
		conn.Close()
		fmt.Printf("%-22s ok\n", t)
	}
	fmt.Println("\nA socket bound to 127.0.0.1 only accepts traffic addressed to 127.0.0.1.")
	fmt.Println("No firewall rule can change that: fix the bind address, not the firewall.")
}

$ go run ./bindlab conflict
second listen: listen tcp 127.0.0.1:7300: bind: address already in use
EADDRINUSE?    true

$ go run ./bindlab loopback
127.0.0.1:7302         ok
192.168.1.2:7302       FAIL  dial tcp 192.168.1.2:7302: connect: connection refused
127.0.0.1:7303         ok
192.168.1.2:7303       ok

$ go run ./bindlab reuseport          # on Linux
connections accepted per listener: [103 105 104 88]
$ go run ./bindlab reuseport          # on macOS
connections accepted per listener: [0 0 0 400]

What to notice:

• "Refused" from your own LAN address. The 127.0.0.1 listener exists, but the connection was addressed to 192.168.1.2:7302, where nothing listens. This is the most common "works on my machine" bug in container and VM setups: the app binds 127.0.0.1 and everything outside the host gets refused. Fix the bind, not the firewall.
• ListenConfig.Control runs your code on the raw file descriptor after socket() but before bind(). That's the only moment socket options like SO_REUSEPORT can be set. This hook is how Go programs reach any socket option the net package doesn't wrap.
• The same code behaves differently on Linux and macOS. Linux load-balances new connections across SO_REUSEPORT sockets by hashing the 4-tuple. BSD-derived systems historically deliver to one socket. Always test kernel behaviour on the kernel you deploy to.

A real incident this explains: during testing for this guide, a backend bound to 127.0.0.1:8081 was killed. Instead of getting "connection refused", the clients started receiving 404s. Another program (Docker Desktop) was listening on the wildcard *:8081. The kernel gives each connection to the most specific matching listener. While the backend was alive, it was more specific. Once it died, the wildcard listener got the traffic. If you see "wrong service answered", check ss -ltnp / lsof -nP -iTCP -sTCP:LISTEN for overlapping binds.

Exercises:

1. Start python3 -m http.server 8000 --bind 0.0.0.0, then try to net.Listen("tcp", "127.0.0.1:8000") in Go. Does it conflict? (On Linux it does: a wildcard bind covers every address.)
2. Run reuseport under strace -f -e trace=setsockopt,bind,listen and find the four setsockopt(... SO_REUSEPORT ...) calls.
3. Read how Linux's Chapter 37 on network views shows these listeners with ss -ltnp, then find which process owns each.

Practice (20 min)

A. Send UDP by hand:

# Terminal 1: listen
nc -u -l 9999

# Terminal 2: send
nc -u 127.0.0.1 9999
# type lines; they appear in terminal 1

Notice: unlike the TCP version in Chapter 19, there's no connection. Kill terminal 1 and keep typing in terminal 2 — it doesn't complain. The data just goes nowhere. That's UDP.

B. See UDP in the wild. Your machine is doing this constantly:

sudo tcpdump -i any -n udp

Leave it a minute. You'll see DNS (port 53), mDNS (5353, .local discovery), maybe NTP (123), DHCP (67/68), and QUIC (443/UDP — that's HTTP/3, Chapter 30).

C. Compare TCP and UDP DNS:

sudo tcpdump -i any -n 'port 53' &
dig example.com                # uses UDP -- 2 packets
dig +tcp example.com           # forces TCP -- handshake + query + teardown

Count the packets in each case. That's the cost of a connection, visible.

D. Check UDP loss statistics on your machine:

netstat -su | head -20      # look for "packet receive errors"
nstat -az | grep -i udp     # Linux

Because UDP loss is silent, these counters are the only way to see it.

Build it in Go (40 min) — measure what UDP doesn't promise

The "What your app must do itself" list above is easier to remember once you've had to implement it. This lab is an echo server plus a client that stamps every datagram with a sequence number, then measures loss, reordering, duplication, RTT percentiles, and jitter. These are the exact metrics a video-call or game client tracks.

// udpping: a UDP echo server and a client that measures what UDP does NOT
// promise you: loss, reordering, duplication, and jitter.
//
//	go run ./udpping -serve :9999
//	go run ./udpping -to 127.0.0.1:9999 -n 200 -every 10ms
package main

import (
	"encoding/binary"
	"errors"
	"flag"
	"fmt"
	"math"
	"net"
	"os"
	"slices"
	"syscall"
	"time"
)

func main() {
	serve := flag.String("serve", "", "run the echo server on this address")
	to := flag.String("to", "", "server address to probe")
	n := flag.Int("n", 100, "number of probes")
	every := flag.Duration("every", 20*time.Millisecond, "interval between probes")
	size := flag.Int("size", 64, "datagram size in bytes (try 1472 and 1473 with a 1500 MTU)")
	flag.Parse()
	switch {
	case *serve != "":
		server(*serve)
	case *to != "":
		client(*to, *n, *every, *size)
	default:
		flag.Usage()
		os.Exit(2)
	}
}

func server(addr string) {
	pc, err := net.ListenPacket("udp", addr)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	fmt.Println("echoing UDP on", pc.LocalAddr())
	buf := make([]byte, 65535)
	for {
		n, from, err := pc.ReadFrom(buf)
		if err != nil {
			continue
		}
		pc.WriteTo(buf[:n], from) // no connection, no state: reply to whoever sent it
	}
}

func client(addr string, n int, every time.Duration, size int) {
	conn, err := net.Dial("udp", addr) // "connected" UDP: the kernel filters replies for us
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	defer conn.Close()
	if size < 16 {
		size = 16
	}

	sent := make(map[uint64]time.Time, n)
	var rtts []time.Duration
	var lastSeq uint64
	received, reordered, dups := 0, 0, 0
	seen := make(map[uint64]bool, n)

	// Receiver goroutine: every reply carries the sequence number we sent.
	type reply struct {
		seq uint64
		at  time.Time
	}
	replies := make(chan reply, n)
	go func() {
		buf := make([]byte, 65535)
		warned := false
		for {
			m, err := conn.Read(buf)
			if errors.Is(err, syscall.ECONNREFUSED) {
				// UDP has no handshake, so how can it be "refused"? The server's
				// kernel answered with ICMP "port unreachable" and ours relayed it.
				if !warned {
					fmt.Println("read: connection refused -> ICMP port unreachable: nothing listens there")
					warned = true
				}
				continue
			}
			if err != nil {
				return // socket closed
			}
			if m >= 8 {
				replies <- reply{binary.BigEndian.Uint64(buf), time.Now()}
			}
		}
	}()

	payload := make([]byte, size)
	tick := time.NewTicker(every)
	defer tick.Stop()
	for seq := uint64(1); seq <= uint64(n); seq++ {
		<-tick.C
		binary.BigEndian.PutUint64(payload, seq)
		sent[seq] = time.Now()
		if _, err := conn.Write(payload); err != nil {
			fmt.Printf("seq %d: write error: %v\n", seq, err) // e.g. "message too long" past the MTU
		}
	drain:
		for {
			select {
			case r := <-replies:
				switch {
				case seen[r.seq]:
					dups++
				case r.seq < lastSeq:
					reordered++
					fallthrough
				default:
					seen[r.seq] = true
					received++
					rtts = append(rtts, r.at.Sub(sent[r.seq]))
					lastSeq = max(lastSeq, r.seq)
				}
			default:
				break drain
			}
		}
	}
	time.Sleep(time.Second) // let stragglers arrive
	for len(replies) > 0 {
		r := <-replies
		if !seen[r.seq] {
			seen[r.seq] = true
			received++
			rtts = append(rtts, r.at.Sub(sent[r.seq]))
		}
	}

	loss := 100 * float64(n-received) / float64(n)
	fmt.Printf("%d sent, %d received, %.1f%% loss, %d reordered, %d duplicated\n",
		n, received, loss, reordered, dups)
	if len(rtts) == 0 {
		return
	}
	slices.Sort(rtts)
	var sum, jitter float64
	for i, r := range rtts {
		sum += float64(r)
		if i > 0 {
			jitter += math.Abs(float64(r - rtts[i-1]))
		}
	}
	fmt.Printf("rtt min/p50/p99/max = %v / %v / %v / %v   avg=%v\n",
		rtts[0], rtts[len(rtts)/2], rtts[len(rtts)*99/100], rtts[len(rtts)-1],
		time.Duration(sum/float64(len(rtts))))
}

$ go run ./udpping -serve 127.0.0.1:9999 &
$ go run ./udpping -to 127.0.0.1:9999 -n 100 -every 2ms
100 sent, 100 received, 0.0% loss, 0 reordered, 0 duplicated
rtt min/p50/p99/max = 52.792µs / 132.083µs / 1.360292ms / 1.360292ms   avg=144.76µs

$ go run ./udpping -to 127.0.0.1:9998 -n 3      # nothing listening
read: connection refused -> ICMP port unreachable: nothing listens there
3 sent, 0 received, 100.0% loss, 0 reordered, 0 duplicated

$ go run ./udpping -to 127.0.0.1:9999 -n 3 -size 70000
seq 1: write error: ... write: message too long

What to notice:

• "Connection refused" on a protocol without connections. The server's kernel answered with an ICMP port unreachable (Chapter 14). Because the client socket is "connected", the kernel hands that error back to the next Read. An unconnected socket (ListenPacket) would never see it.
• message too long: a UDP datagram can't exceed 65,507 bytes of payload, and anything over the path MTU is fragmented at the IP layer (Chapter 15). Real UDP protocols (DNS, QUIC) keep datagrams around 1,200 bytes to avoid fragmentation completely.
• One goroutine sends, another receives. UDP gives you no request/response pairing. The sequence number is your pairing, and the same is true of QUIC packet numbers and RTP in voice calls.

Exercises:

1. (Linux) Add impairment and measure it: sudo tc qdisc add dev lo root netem delay 20ms 5ms loss 2% reorder 10%, run the client against 127.0.0.1, then remove it with sudo tc qdisc del dev lo root. Do your numbers match what you configured?
2. Find the largest -size that works between two real machines on your LAN. It should be 1472 bytes (1500 MTU − 20 IP − 8 UDP) unless you add a don't-fragment option.
3. Add a retransmit: if a reply doesn't arrive within 3× the median RTT, resend it. Congratulations, you've started reinventing TCP. Stop and read Chapter 22.

Practice (30 min) — capture your first handshake

This is the most important practice exercise in Part 5.

# Terminal 1: capture ONLY handshake packets to a site
sudo tcpdump -i any -n 'tcp port 443 and host example.com'

# Terminal 2:
curl -s https://example.com > /dev/null

You will see, right at the top:

IP 192.168.1.50.51000 > 93.184.216.34.443: Flags [S],  seq 1234567, win 64240,
      options [mss 1460,sackOK,TS val ...,nop,wscale 7], length 0
IP 93.184.216.34.443 > 192.168.1.50.51000: Flags [S.], seq 7654321, ack 1234568,
      win 65535, options [mss 1460,sackOK,...], length 0
IP 192.168.1.50.51000 > 93.184.216.34.443: Flags [.],  ack 7654322, win 502, length 0

Read it:

• [S] = SYN. [S.] = SYN+ACK (the . means ACK). [.] = ACK only.
• seq 1234567 = "my numbering starts here"
• ack 1234568 = "I got your 1234567, send me 1234568 next" (note: +1)
• options [mss 1460, sackOK, wscale 7] = the negotiation described above

Congratulations — you have now seen the foundation of the internet with your own eyes. Save this capture in notes.md.

Now produce the other two outcomes:

# CONNECTION REFUSED -- connect to a port with nothing on it, locally
sudo tcpdump -i any -n 'tcp port 9999' &
nc -v 127.0.0.1 9999
# you'll see: SYN then RST, and nc says "Connection refused"

# TIMEOUT -- connect to a filtered port on the internet
sudo tcpdump -i any -n 'host example.com and tcp port 81' &
nc -v -w 5 example.com 81
# you'll see: SYN ... SYN ... SYN (retries), no answer

Watch the retry timing on the timeout case: roughly 1 s, then 2 s, then 4 s. That's exponential backoff, and it's why a firewalled port takes so long to fail while a refused port fails instantly.

Build it in Go (20 min) — a handshake outcome classifier

The three outcomes above (connected, refused, timed out) each surface as a different error value in code. Production code that treats them all as "error" throws away the most useful diagnostic information it has. This program tells them apart:

// dialcheck: try a TCP connection and say WHICH of the handshake outcomes
// from Chapter 21 happened -- connected, refused, timed out, or something else.
//
//	go run ./dialcheck example.com:443 127.0.0.1:9 example.com:81 nosuchhost.invalid:80
package main

import (
	"context"
	"errors"
	"fmt"
	"net"
	"os"
	"syscall"
	"time"
)

func main() {
	for _, target := range os.Args[1:] {
		fmt.Printf("%-28s %s\n", target, check(target, 3*time.Second))
	}
}

func check(target string, timeout time.Duration) string {
	ctx, cancel := context.WithTimeout(context.Background(), timeout)
	defer cancel()

	start := time.Now()
	var d net.Dialer
	conn, err := d.DialContext(ctx, "tcp", target)
	took := time.Since(start).Round(time.Millisecond)
	if err == nil {
		defer conn.Close()
		return fmt.Sprintf("CONNECTED in %v  (%s -> %s)", took, conn.LocalAddr(), conn.RemoteAddr())
	}

	var dnsErr *net.DNSError
	var netErr net.Error
	switch {
	case errors.As(err, &dnsErr):
		return fmt.Sprintf("DNS FAILURE (never sent a SYN): %v", dnsErr.Err)
	case errors.Is(err, syscall.ECONNREFUSED):
		return fmt.Sprintf("REFUSED in %v: host answered with RST -> nothing listening", took)
	case errors.Is(err, syscall.EHOSTUNREACH), errors.Is(err, syscall.ENETUNREACH):
		return fmt.Sprintf("UNREACHABLE in %v: no route, or an ICMP unreachable came back", took)
	case errors.As(err, &netErr) && netErr.Timeout():
		return fmt.Sprintf("TIMED OUT after %v: SYNs got no answer -> filtered/dropped", took)
	default:
		return fmt.Sprintf("OTHER after %v: %v", took, err)
	}
}

$ go run ./dialcheck example.com:443 127.0.0.1:9 example.com:81 nosuchhost.invalid:80
example.com:443              CONNECTED in 93ms  (192.168.1.2:62150 -> 172.66.147.243:443)
127.0.0.1:9                  REFUSED in 0s: host answered with RST -> nothing listening
example.com:81               TIMED OUT after 3s: SYNs got no answer -> filtered/dropped
nosuchhost.invalid:80        DNS FAILURE (never sent a SYN): no such host

What to notice:

• errors.Is(err, syscall.ECONNREFUSED) works because Go wraps the kernel's error number inside *net.OpError → *os.SyscallError. errors.Is and errors.As unwrap the chain. See the Go guide's error-handling chapter for the pattern.
• Connect time ≈ 1 RTT. The 93 ms "CONNECTED" figure is the handshake you captured above. Your app can measure RTT without ICMP at all, which is useful where ping is blocked.
• The timeout is the caller's decision. Without the context.WithTimeout, the kernel would keep retrying SYNs for about 2 minutes on Linux (net.ipv4.tcp_syn_retries = 6). Always put a deadline on dials.

Exercises:

1. Lower the timeout to 500 ms and dial example.com:81 again. Then raise it to 10 s and count the SYN retransmissions with tcpdump. Do they follow 1 s, 2 s, 4 s?
2. Make check run on a list of targets concurrently with a sync.WaitGroup. A filtered port shouldn't hold up the rest. Chapter 53's policytest lab is this idea grown up.
3. Add an IPv6-vs-IPv4 column: dial tcp4 and tcp6 separately. Hosts whose IPv6 path is broken are why browsers use Happy Eyeballs (racing both).

Across the series: the Go plan's Day 32 TCP echo server builds the server side of this handshake, and Day 52 uses these states to debug CLOSE_WAIT leaks (see also Chapter 23 here).

Practice (30 min)

A. See sequence numbers and ACKs in a real transfer:

sudo tcpdump -i any -n -c 40 'tcp port 443 and host example.com'
# in another terminal:
curl -s https://example.com > /dev/null

Follow the seq and ack numbers down the capture. Watch seq climb by the length of each data packet, and ack climb to match.

B. Confirm SACK was negotiated — look at the SYN packets from Chapter 21 for sackOK. Then check your system's setting:

sysctl net.ipv4.tcp_sack        # Linux; should be 1

C. Cause real retransmissions and watch them. (Linux only; this deliberately degrades your network for a moment — do it on a test machine or be ready to undo.)

# Add 10% packet loss to outgoing traffic
sudo tc qdisc add dev eth0 root netem loss 10%

# Now download something and capture
sudo tcpdump -i eth0 -n 'host example.com' -w /tmp/loss.pcap &
curl -s https://example.com > /dev/null
sudo pkill tcpdump

# IMPORTANT: remove the impairment
sudo tc qdisc del dev eth0 root

Open /tmp/loss.pcap in Wireshark. It will highlight retransmissions and duplicate ACKs in black/red. You have just watched TCP heal itself.

D. Check your machine's retransmission counters:

# Linux
nstat -az | grep -Ei 'retrans|TCPLostRetransmit|TCPTimeouts'
# macOS
netstat -s -p tcp | grep -i retrans

A small number of retransmissions is normal. A rising percentage of segments retransmitted (over ~1%) indicates real trouble.

Build it in Go (25 min) — TCP is a byte stream, not a message stream

Everything in this chapter (sequence numbers, ACKs, retransmissions) works on bytes, not on your messages. TCP doesn't know or care where one of your Write calls ended and the next began. This surprises nearly every developer the first time they write a protocol by hand:

// stream: prove that TCP is a byte stream with no message boundaries, then fix
// it with length-prefix framing.
//
//	go run ./stream
package main

import (
	"bufio"
	"encoding/binary"
	"fmt"
	"io"
	"net"
	"time"
)

var messages = []string{"hello", "how are you?", "bye"}

func main() {
	ln, err := net.Listen("tcp", "127.0.0.1:0") // port 0: let the kernel pick a free port
	if err != nil {
		panic(err)
	}
	defer ln.Close()

	fmt.Println("== naive: one Write per message, one Read per message?")
	go naiveSender(ln.Addr().String())
	naiveReceiver(ln)

	fmt.Println("\n== framed: 4-byte length prefix before every message")
	go framedSender(ln.Addr().String())
	framedReceiver(ln)
}

func naiveSender(addr string) {
	c, _ := net.Dial("tcp", addr)
	defer c.Close()
	for _, m := range messages {
		c.Write([]byte(m)) // three separate Write calls...
	}
}

func naiveReceiver(ln net.Listener) {
	c, _ := ln.Accept()
	defer c.Close()
	time.Sleep(50 * time.Millisecond) // let all three writes land in the receive buffer
	buf := make([]byte, 1024)
	for {
		n, err := c.Read(buf)
		if err != nil {
			return
		}
		fmt.Printf("Read returned %2d bytes: %q\n", n, buf[:n]) // ...usually ONE Read
	}
}

func framedSender(addr string) {
	c, _ := net.Dial("tcp", addr)
	defer c.Close()
	w := bufio.NewWriter(c)
	for _, m := range messages {
		binary.Write(w, binary.BigEndian, uint32(len(m)))
		w.WriteString(m)
	}
	w.Flush() // one syscall for everything: framing makes batching safe
}

func framedReceiver(ln net.Listener) {
	c, _ := ln.Accept()
	defer c.Close()
	r := bufio.NewReader(c)
	for {
		var size uint32
		if err := binary.Read(r, binary.BigEndian, &size); err != nil {
			return // io.EOF: the sender closed cleanly
		}
		if size > 1<<20 {
			fmt.Println("refusing absurd frame size", size) // never trust a length from the wire
			return
		}
		msg := make([]byte, size)
		if _, err := io.ReadFull(r, msg); err != nil { // ReadFull loops over short reads
			fmt.Println("truncated frame:", err)
			return
		}
		fmt.Printf("message: %q\n", msg)
	}
}

$ go run ./stream
== naive: one Write per message, one Read per message?
Read returned 20 bytes: "hellohow are you?bye"

== framed: 4-byte length prefix before every message
message: "hello"
message: "how are you?"
message: "bye"

What to notice:

• Three writes arrived as one read. The bytes were all in the receive buffer by the time the reader looked, so one Read returned all of them. On a slow or lossy network the opposite also happens: one write can arrive across several reads. Both are correct TCP behaviour.
• io.ReadFull keeps reading until the buffer is full. Read alone may legally return fewer bytes, and assuming it won't is a classic production bug that only shows up under load or on bad networks.
• Validate lengths that come from the wire. A peer that sends a length of 4 GB would otherwise make you allocate 4 GB. The 1<<20 check is a one-line denial-of-service fix.
• Real protocols choose either delimiters (HTTP/1.1 headers end with \r\n\r\n; Redis uses \r\n) or length prefixes (HTTP/2 frames, gRPC messages, TLS records, Postgres, Kafka). Chapter 39 shows what happens when two parsers disagree about framing.

Exercises:

1. Remove the time.Sleep in naiveReceiver and run it 20 times. Does the split change from run to run? (It depends on timing, which is exactly the problem.)
2. Switch the framed version to newline delimiters with bufio.Scanner. Then send a message that contains a newline and watch it break. That's why binary protocols use length prefixes.
3. The Go guide's framing section and the Go plan's binary KV protocol (Day 53) take this to a full protocol with versioning and message types. Build the KV server.

Practice (25 min)

A. Watch a full close:

sudo tcpdump -i any -n 'tcp port 443 and host example.com'
curl -s https://example.com > /dev/null

At the end of the capture, find the [F.] (FIN+ACK) packets and the final [.].

B. Watch a reset:

sudo tcpdump -i any -n 'tcp port 9999' &
nc -v 127.0.0.1 9999          # nothing is listening

You'll see [R.] — an RST.

C. See the state histogram on your own machine:

ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

Then open a lot of browser tabs and run it again.

D. Create TIME_WAIT deliberately:

for i in $(seq 1 20); do curl -s -o /dev/null http://example.com; done
ss -tan state time-wait | wc -l
# wait 60 seconds, run again -- they disappear

E. Find any CLOSE_WAIT on your system (and identify the guilty program):

ss -tanp state close-wait          # Linux
lsof -i -sTCP:CLOSE_WAIT           # macOS/Linux

If you find some that persist, you've found a real bug in some software.

Build it in Go (30 min) — manufacture a CLOSE_WAIT leak, then fix it

TIME_WAIT is normal. CLOSE_WAIT that keeps growing is always a bug in your program: the peer has closed, the kernel has acknowledged its FIN, and the socket is waiting for your code to call close(). It will wait forever. Each leaked socket holds a file descriptor, and eventually the process hits its open-files limit and stops accepting anything (too many open files).

// closewait: manufacture the CLOSE_WAIT leak from Chapter 23, watch it with
// ss/lsof, then fix it -- and see a clean half-close with CloseWrite.
//
//	go run ./closewait -leak     # server forgets to Close
//	go run ./closewait           # server closes properly
package main

import (
	"bufio"
	"flag"
	"fmt"
	"io"
	"net"
	"os"
	"time"
)

func main() {
	leak := flag.Bool("leak", false, "simulate the bug: never close accepted connections")
	flag.Parse()

	ln, err := net.Listen("tcp", "127.0.0.1:7070")
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	fmt.Printf("server pid %d on %s\n", os.Getpid(), ln.Addr())
	fmt.Println("in another terminal run:")
	fmt.Println("  Linux:  watch -n1 \"ss -tan state close-wait '( sport = :7070 )'\"")
	fmt.Println("  macOS:  watch -n1 \"netstat -an -p tcp | grep 7070\"")

	go func() {
		for {
			c, err := ln.Accept()
			if err != nil {
				return
			}
			go handle(c, *leak)
		}
	}()

	// Ten clients connect, send a request, half-close, read the reply, leave.
	for i := 0; i < 10; i++ {
		c, err := net.Dial("tcp", ln.Addr().String())
		if err != nil {
			fmt.Println(err)
			continue
		}
		fmt.Fprintf(c, "request %d\n", i)
		c.(*net.TCPConn).CloseWrite()                   // send FIN: "I'm done talking, but still listening"
		reply, _ := bufio.NewReader(c).ReadString('\n') // the server can still answer after our FIN
		fmt.Printf("client %d got %q\n", i, reply)
		c.Close()
	}
	fmt.Println("clients finished. Server stays up 60s so you can look; Ctrl-C to quit.")
	time.Sleep(60 * time.Second)
}

func handle(c net.Conn, leak bool) {
	req, _ := io.ReadAll(c) // returns when the client's FIN arrives (EOF)
	fmt.Fprintf(c, "ok: %s", req)
	if leak {
		// BUG: no c.Close(). The client's FIN was acknowledged, so the kernel
		// moves our side to CLOSE_WAIT and waits for US to close. It will wait
		// forever, holding a file descriptor and kernel memory.
		return
	}
	c.Close()
}

Real output on Linux:

$ go run ./closewait -leak &
$ ss -tan state close-wait '( sport = :7070 )'
Recv-Q Send-Q Local Address:Port Peer Address:Port
0      0          127.0.0.1:7070    127.0.0.1:55228
0      0          127.0.0.1:7070    127.0.0.1:55258
...                                                  <- 10 sockets, and they never go away

$ go run ./closewait            # the fixed version
$ ss -tan state close-wait '( sport = :7070 )' | tail -n +2 | wc -l
0

What to notice:

• The client side is in FIN_WAIT_2 (run netstat -an | grep 7070 and you'll see both halves). The client sent its FIN and got an ACK, and is now waiting for the server's FIN, which never comes.
• CloseWrite() is a half-close: it sends FIN ("I've finished sending") but keeps reading. The server's io.ReadAll returns when that FIN arrives, and the server can still reply. HTTP/1.0 used exactly this pattern, and so do nc -N and many proxies.
• In Go, the fix is almost always a missing defer c.Close(), often on an error path, or a resp.Body from net/http that was never closed. That last one leaks the same way on the client side.

Exercises:

1. While -leak runs, check the file descriptors: ls /proc/$(pgrep closewait)/fd | wc -l. Then lower the limit with ulimit -n 64 and change the loop to 100 clients. What error do you get? (Linux guide, Chapter 43 covers ulimit.)
2. Write the same bug with net/http: a client that does resp, _ := http.Get(url) and never closes resp.Body. Watch the connections pile up in ESTABLISHED instead. Which side is now leaking?
3. The Go plan's Day 52 exercise has a production-style version with a leak detector. Do it next.

Practice (25 min)

A. See the window in your own traffic:

sudo tcpdump -i any -n 'tcp port 443' -c 20

Look at the win value on each packet. After the handshake, the numbers are scaled — multiply by 2^(scale factor from the handshake) for the real value.

B. Look at live connection details (Linux — this is a great command):

ss -tin

Look for wscale:7,7 (window scaling active), rtt: (measured round-trip time), cwnd: (Chapter 25), and send X Mbps (its own throughput estimate).

C. Prove the window/RTT formula. Compare a download from something nearby with something far away:

curl -o /dev/null -s -w 'speed: %{speed_download} bytes/s   time: %{time_total}s\n' \
  https://speed.cloudflare.com/__down?bytes=10000000

Then compare ping times to servers at different distances. The relationship between latency and achievable throughput will be visible.

D. Check your system's buffer limits:

sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.core.rmem_max   # Linux
sysctl net.inet.tcp.recvspace net.inet.tcp.sendspace           # macOS

The third number in tcp_rmem is the maximum receive buffer. If it's smaller than your BDP, that's your speed limit.

Build it in Go (25 min) — drive the window to zero

Flow control is easiest to understand from the sender's side: when the receiver stops reading, the sender's Write stops returning. This lab connects a fast writer to a deliberately slow reader (4 KB every 500 ms) with small buffers, so the window fills within a second.

// zerowindow: a receiver that reads slowly, so you can watch flow control
// (Chapter 24) fill the receive buffer, shrink the window to zero, and make
// the sender's Write block.
//
//	go run ./zerowindow
//	# meanwhile:  ss -tnoi '( sport = :7071 or dport = :7071 )'   (Linux)
//	#             netstat -an -p tcp | grep 7071                   (macOS: watch Recv-Q / Send-Q)
package main

import (
	"fmt"
	"net"
	"time"
)

func main() {
	ln, err := net.Listen("tcp", "127.0.0.1:7071")
	if err != nil {
		panic(err)
	}

	go func() { // the SLOW receiver: 4 KB every 500 ms
		c, _ := ln.Accept()
		c.(*net.TCPConn).SetReadBuffer(64 * 1024) // a small window, so it fills fast
		buf := make([]byte, 4096)
		for {
			time.Sleep(500 * time.Millisecond)
			if _, err := c.Read(buf); err != nil {
				return
			}
		}
	}()

	c, err := net.Dial("tcp", ln.Addr().String())
	if err != nil {
		panic(err)
	}
	c.(*net.TCPConn).SetWriteBuffer(64 * 1024)

	chunk := make([]byte, 16*1024)
	total := 0
	for i := 0; i < 64; i++ {
		start := time.Now()
		n, err := c.Write(chunk) // returns once the bytes are in OUR send buffer
		if err != nil {
			fmt.Println(err)
			return
		}
		total += n
		if wait := time.Since(start); wait > 50*time.Millisecond {
			fmt.Printf("write #%-2d BLOCKED %-6v (total %4d KB) <- receiver's window is full\n",
				i, wait.Round(time.Millisecond), total/1024)
		} else {
			fmt.Printf("write #%-2d instant        (total %4d KB)\n", i, total/1024)
		}
	}
}

The program's own output shows writes going from instant to blocked. The kernel's view is the interesting part (Linux, during the run):

$ ss -tnoi '( sport = :7071 or dport = :7071 )'
ESTAB 79872  0        127.0.0.1:7071   127.0.0.1:51062           <- receiver: 78 KB unread (Recv-Q)
ESTAB 0      128000   127.0.0.1:51062  127.0.0.1:7071  timer:(persist,701ms,0)
     ... backoff:4 ... rwnd_limited:6008ms(100.0%) ... notsent:128000

How to read it:

• Receiver Recv-Q 79872: bytes the kernel holds that the application hasn't Read yet. The window it advertises shrinks by exactly this amount.
• Sender Send-Q 128000 / notsent:128000: bytes the sender has accepted from Write but can't send, because the window is closed.
• timer:(persist,...): the zero-window probe timer from this chapter. The sender periodically asks "is there room yet?", backing off each time (backoff:4).
• rwnd_limited:...(100.0%): Linux is telling you directly that the whole connection was limited by the receiver's window, not by the network. When you see this in production, look at the receiving application, not the network.

Exercises:

1. Change the reader's sleep to 0 and watch every write become instant. The network didn't change; the application did.
2. Remove the SetReadBuffer/SetWriteBuffer calls. Linux autotunes the buffers upwards, so it takes longer to block. How many KB get through before the first blocked write?
3. In a real service this looks like a consumer that's too slow, for example a log shipper behind a slow disk. Find the equivalent in the Linux guide's I/O stack chapter: a slow disk produces exactly this kind of backpressure.

Practice (25 min)

A. See your congestion window live:

# Linux: start a big download, then watch in another terminal
curl -o /dev/null https://speed.cloudflare.com/__down?bytes=100000000 &
watch -n0.5 "ss -tin | grep -A1 cloudflare | head -4"

Watch cwnd: grow. That's slow start happening in front of you.

B. Check which algorithm you're using:

sysctl net.ipv4.tcp_congestion_control          # Linux
sysctl net.ipv4.tcp_available_congestion_control

C. Watch the sawtooth. Add a little loss and download something, then graph throughput in Wireshark (Statistics → I/O Graph):

sudo tc qdisc add dev eth0 root netem loss 0.5%
curl -o /dev/null https://speed.cloudflare.com/__down?bytes=50000000
sudo tc qdisc del dev eth0 root          # UNDO

D. Prove that parallel connections beat one. This is a striking demo:

# one connection
time curl -s -o /dev/null https://speed.cloudflare.com/__down?bytes=50000000

# four connections in parallel
time ( for i in 1 2 3 4; do
    curl -s -o /dev/null https://speed.cloudflare.com/__down?bytes=12500000 &
  done; wait )

Same total bytes. On a long or slightly lossy path, the parallel version is often much faster. That is congestion control, measured.

Practice (20 min)

A. Check whether your tools set it. Many do:

# Linux -- watch a program's socket options as it runs
strace -f -e trace=setsockopt curl -s https://example.com 2>&1 | grep -i nodelay

B. Measure a suspiciously round latency. If you have a local service, time many small requests:

for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{time_total}\n' http://localhost:8000/
done | sort -n

Values clustered at 0.04 s with a local server = the fingerprint.

C. Recognise the sibling problem. Also watch for delays that are exact multiples of common timers: 200 ms (retransmission timeout, Chapter 22), 1 s / 3 s / 7 s (SYN retries, Chapter 21), 5 s (a dead DNS server, Chapter 18). Suspiciously round numbers in latency are always a timer, never a coincidence.

Build it in Go (20 min) — the 40 ms mystery, on demand

In Go, the fix above is already applied. Every TCP connection Go creates has TCP_NODELAY set by default. That's why Go services rarely hit this bug, and why it's confusing when they talk to a peer written in something else. To see the problem, this lab turns Nagle back on with SetNoDelay(false) and uses the classic "write-write-read" pattern:

// nagle: reproduce Chapter 26's "40 ms mystery" on purpose.
//
// Go sets TCP_NODELAY on every TCP connection by default (Nagle OFF), so
// Go programs rarely hit this -- unless someone turns Nagle back on, or the
// peer is a program in another language. Here we turn it on deliberately.
//
//	go run ./nagle
package main

import (
	"bufio"
	"fmt"
	"net"
	"slices"
	"time"
)

func main() {
	ln, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		panic(err)
	}
	go echoServer(ln)

	for _, noDelay := range []bool{true, false} {
		c, err := net.Dial("tcp", ln.Addr().String())
		if err != nil {
			panic(err)
		}
		tc := c.(*net.TCPConn)
		tc.SetNoDelay(noDelay) // false = Nagle's algorithm ON
		r := bufio.NewReader(c)

		var lat []time.Duration
		for i := 0; i < 20; i++ {
			start := time.Now()
			// The classic bad pattern: header and body in TWO small writes,
			// then wait for a reply ("write-write-read").
			c.Write([]byte("HEADER:"))
			c.Write([]byte("body\n"))
			if _, err := r.ReadString('\n'); err != nil {
				panic(err)
			}
			lat = append(lat, time.Since(start))
		}
		c.Close()
		slices.Sort(lat)
		fmt.Printf("TCP_NODELAY=%-5v  p50=%-10v max=%v\n", noDelay, lat[len(lat)/2], lat[len(lat)-1])
	}
	fmt.Println("\nIf NODELAY=false is ~40ms (Linux) or higher, you just met Nagle + delayed ACK.")
	fmt.Println("Fix: one write per message (bufio.Writer + Flush), or keep NODELAY on.")
}

func echoServer(ln net.Listener) {
	for {
		c, err := ln.Accept()
		if err != nil {
			return
		}
		go func() {
			defer c.Close()
			r := bufio.NewReader(c)
			for {
				line, err := r.ReadString('\n')
				if err != nil {
					return
				}
				c.Write([]byte(line))
			}
		}()
	}
}

$ go run ./nagle                  # Linux
TCP_NODELAY=true   p50=5.709µs    max=49.25µs
TCP_NODELAY=false  p50=43.952667ms max=46.074958ms

Four orders of magnitude, caused by one socket option and two small writes. On macOS loopback the effect often doesn't appear, because delayed ACK behaves differently there. Run it on Linux (or in the container from Section 0.8) to see the real fingerprint.

What to notice:

• The fix in the output, "one write per message", is the cleanest one. bufio.Writer collects the header and body and Flush() sends them in a single write(), so Nagle never has a small packet to hold back.
• SetNoDelay is a thin wrapper around setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, ...), the same call the Python snippet above makes.

Exercises:

1. Keep SetNoDelay(false) but wrap the connection in a bufio.Writer and Flush() after the body. Does the 40 ms disappear?
2. Run the server under strace -e trace=setsockopt and confirm Go set TCP_NODELAY on the accepted connection too.
3. Throughput is 1 / 44 ms ≈ 23 requests per second per connection. Find the "suspiciously round number" in that.

Practice (25 min)

A. Measure your own physics:

ping -c 10 <your router>          # ~1 ms
ping -c 10 1.1.1.1                 # your ISP + a bit
ping -c 10 <something far away>    # distance

Divide the far one by 2 and multiply by 200 km/ms to estimate the fibre distance. Compare with a map. It's usually surprisingly close.

B. Do the bufferbloat test above. Record your idle and loaded latency in notes.md.

C. Prove bandwidth ≠ speed for web pages:

curl -w 'dns %{time_namelookup}  tcp %{time_connect}  tls %{time_appconnect}  ttfb %{time_starttransfer}  total %{time_total}\n' \
     -o /dev/null -s https://example.com

Notice how much of total happens before any content arrives. That portion is pure latency and is unaffected by your bandwidth.

D. Check your current queue discipline (Linux):

tc qdisc show

If it says pfifo_fast, you have a dumb FIFO queue. fq_codel or cake is what you want.

Practice (30 min) — speak HTTP by hand

A. Type HTTP yourself. This makes the whole chapter click:

# Connect to a web server manually (unencrypted port 80)
nc example.com 80

Then type exactly this, including the blank line at the end:

GET / HTTP/1.1
Host: example.com
Connection: close

(Press Enter twice.) The server replies with headers and HTML. You just were a web browser.

B. The encrypted version (port 443 needs TLS, so use openssl):

openssl s_client -connect example.com:443 -servername example.com -quiet
# then type the same GET request

C. Use curl properly — -v shows you everything:

curl -v https://example.com 2>&1 | head -40

Lines starting with > are what you sent; < is what came back; * is curl's own commentary (DNS, TCP, TLS).

D. Explore status codes:

curl -sI https://example.com                    # HEAD -- headers only
curl -sI https://example.com/nothing-here       # 404
curl -sI -L https://google.com                  # -L follows redirects; watch the 301
curl -s -o /dev/null -w '%{http_code}\n' https://example.com

E. See keep-alive save round trips:

# two requests on ONE connection
curl -s -o /dev/null -w 'req1 %{time_total}\n' \
     -o /dev/null -w 'req2 %{time_total}\n' \
     https://example.com https://example.com

The second is much faster — no TCP or TLS handshake needed.

F. Watch HTTP on the wire (use plain HTTP so it's readable):

sudo tcpdump -i any -n -A 'tcp port 80 and host example.com'
# in another terminal:
curl -s http://example.com > /dev/null

-A prints the payload as text. You'll see the request and response in plain sight — a good reminder of why HTTPS exists.

Build it in Go (35 min) — HTTP by hand, over TCP and TLS

nc let you type HTTP. This program does the same, then solves the problem nc hides from you: where does the response body end? HTTP/1.1 has three answers (Content-Length, Transfer-Encoding: chunked, or "when the server closes"), and every HTTP client has to implement all three.

// rawhttp: speak HTTP/1.1 by hand over a TCP (or TLS) connection and decode
// the response framing yourself -- Content-Length vs chunked.
//
//	go run ./rawhttp http://neverssl.com/
//	go run ./rawhttp https://example.com/
package main

import (
	"bufio"
	"crypto/tls"
	"fmt"
	"io"
	"net"
	"net/url"
	"os"
	"strconv"
	"strings"
	"time"
)

func main() {
	if len(os.Args) != 2 {
		fmt.Println("usage: rawhttp URL")
		os.Exit(2)
	}
	u, err := url.Parse(os.Args[1])
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	port := u.Port()
	if port == "" {
		port = map[string]string{"http": "80", "https": "443"}[u.Scheme]
	}
	addr := net.JoinHostPort(u.Hostname(), port)

	// Layer 4: a plain TCP connection...
	conn, err := net.DialTimeout("tcp", addr, 5*time.Second)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	defer conn.Close()
	// ...wrapped in TLS for https. Everything after this line is identical.
	if u.Scheme == "https" {
		tc := tls.Client(conn, &tls.Config{ServerName: u.Hostname(), NextProtos: []string{"http/1.1"}})
		if err := tc.Handshake(); err != nil {
			fmt.Println("TLS:", err)
			os.Exit(1)
		}
		conn = tc
	}
	conn.SetDeadline(time.Now().Add(10 * time.Second))

	// Layer 7: an HTTP request is just text with CRLF line endings.
	path := u.RequestURI()
	req := "GET " + path + " HTTP/1.1\r\n" +
		"Host: " + u.Host + "\r\n" + // mandatory in HTTP/1.1: one IP, many sites
		"User-Agent: rawhttp-lab/1.0\r\n" +
		"Accept-Encoding: identity\r\n" + // ask for no compression so we can read it
		"Connection: close\r\n" +
		"\r\n" // the empty line ends the headers
	fmt.Printf(">>> request (%d bytes)\n%s", len(req), req)
	io.WriteString(conn, req)

	r := bufio.NewReader(conn)
	status, _ := r.ReadString('\n')
	fmt.Printf("<<< %s", status)
	headers := map[string]string{}
	for {
		line, err := r.ReadString('\n')
		if err != nil || line == "\r\n" {
			break
		}
		fmt.Print("    ", line)
		k, v, _ := strings.Cut(strings.TrimRight(line, "\r\n"), ":")
		headers[strings.ToLower(k)] = strings.TrimSpace(v)
	}

	// Framing: how do we know where the body ends? (Chapter 39)
	var body []byte
	switch {
	case strings.EqualFold(headers["transfer-encoding"], "chunked"):
		body, err = readChunked(r)
		fmt.Printf("\nbody framing: chunked -> %d bytes after de-chunking\n", len(body))
	case headers["content-length"] != "":
		n, _ := strconv.Atoi(headers["content-length"])
		body = make([]byte, n)
		_, err = io.ReadFull(r, body)
		fmt.Printf("\nbody framing: Content-Length: %d\n", n)
	default:
		body, err = io.ReadAll(r) // only valid because we sent Connection: close
		fmt.Printf("\nbody framing: read until the server closed (%d bytes)\n", len(body))
	}
	if err != nil {
		fmt.Println("body error:", err)
	}
	fmt.Printf("first 120 bytes: %q\n", body[:min(120, len(body))])
}

// readChunked decodes "<hex size>\r\n<data>\r\n ... 0\r\n\r\n".
func readChunked(r *bufio.Reader) ([]byte, error) {
	var out []byte
	for {
		line, err := r.ReadString('\n')
		if err != nil {
			return out, err
		}
		sizeStr, _, _ := strings.Cut(strings.TrimSpace(line), ";") // ignore chunk extensions
		size, err := strconv.ParseInt(sizeStr, 16, 64)
		if err != nil {
			return out, fmt.Errorf("bad chunk size %q", sizeStr)
		}
		if size == 0 {
			return out, nil // (trailers would follow; we ignore them)
		}
		chunk := make([]byte, size)
		if _, err := io.ReadFull(r, chunk); err != nil {
			return out, err
		}
		out = append(out, chunk...)
		r.Discard(2) // the CRLF after each chunk
	}
}

$ go run ./rawhttp http://example.com/
>>> request (112 bytes)
GET / HTTP/1.1
Host: example.com
User-Agent: rawhttp-lab/1.0
Accept-Encoding: identity
Connection: close

<<< HTTP/1.1 200 OK
    Content-Type: text/html; charset=utf-8
    Transfer-Encoding: chunked
    Server: cloudflare
    Age: 13523
    cf-cache-status: HIT
    alt-svc: h3=":443"; ma=86400

body framing: chunked -> 577 bytes after de-chunking

What to notice:

• Going from HTTP to HTTPS is a four-line change. tls.Client(conn, ...) wraps the TCP connection, and every line after it is unchanged, because a *tls.Conn is also a net.Conn. That's the layering of Chapter 3, expressed as an interface.
• The response headers tell a story. cf-cache-status: HIT and Age: 13523 mean a CDN answered from cache, a response about 3.75 hours old (Chapter 40). alt-svc: h3=":443" advertises HTTP/3 for next time (Chapter 30).
• Chunked framing exists because a server that streams a response doesn't know its length in advance. Each chunk is <hex length>\r\n<data>\r\n, and a zero-length chunk ends the body.

Exercises:

1. Remove Connection: close and send two requests on the same connection, reading the first response fully (using its framing) before writing the second. You've just implemented keep-alive, the "single biggest performance idea" above.
2. Send a HEAD request and keep the Content-Length branch. Why does it hang? (HEAD responses carry a Content-Length but no body. Clients must know the method to parse the response.)
3. Compare with the real thing: the Go plan's raw HTTP/1.1 server (Day 61) builds the server side, and the HTTPS guide's HTTP chapter covers HTTP/2 and HTTP/3 framing.

Practice (30 min)

A. Inspect a real handshake:

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>&1 | head -50

Look for: Protocol, Cipher, the certificate chain (0 s: is the site, 1 s: is the intermediate), and Verify return code: 0 (ok).

B. Read the certificate itself:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

You'll see who it's for, who signed it, when it expires, and which hostnames it covers.

C. Measure the handshake cost:

curl -w 'tcp: %{time_connect}s   tls: %{time_appconnect}s   (tls cost: computed below)\n' \
     -o /dev/null -s https://example.com

time_appconnect − time_connect is what TLS cost you.

D. Watch the handshake in tcpdump. You'll see the ClientHello and ServerHello in the clear, then everything becomes opaque:

sudo tcpdump -i any -n 'tcp port 443 and host example.com' -c 20
curl -s https://example.com > /dev/null

E. See a certificate failure safely:

curl -v https://expired.badssl.com 2>&1 | grep -i -A3 'certificate'
curl -v https://wrong.host.badssl.com 2>&1 | grep -i 'subject\|SSL'

badssl.com is a site specifically built to demonstrate every kind of TLS failure. Explore it in a browser too — very instructive.

F. Check your own site's configuration (if you have one): use ssllabs.com/ssltest — a free, thorough grading tool.

Build it in Go (30 min) — a scriptable openssl s_client

This program completes a real TLS handshake and prints what was negotiated: version, cipher, ALPN, and the certificate chain with days to expiry. More usefully, it turns the four classic certificate failures into specific diagnoses instead of one generic "x509 error".

// tlsinspect: connect, complete a TLS handshake, and print what was negotiated
// and who the server claims to be -- a scriptable `openssl s_client`.
//
//	go run ./tlsinspect example.com
//	go run ./tlsinspect -sni github.com example.com:443   # SNI the edge has no cert for
//	go run ./tlsinspect expired.badssl.com self-signed.badssl.com wrong.host.badssl.com
package main

import (
	"crypto/tls"
	"crypto/x509"
	"errors"
	"flag"
	"fmt"
	"net"
	"os"
	"strings"
	"time"
)

func main() {
	sni := flag.String("sni", "", "override the SNI/hostname to verify (default: the host)")
	flag.Parse()
	exit := 0
	for _, target := range flag.Args() {
		if !strings.Contains(target, ":") {
			target += ":443"
		}
		if err := inspect(target, *sni); err != nil {
			fmt.Printf("%s\n  FAILED: %s\n\n", target, explain(err))
			exit = 1
		}
	}
	os.Exit(exit)
}

func inspect(target, sni string) error {
	host, _, _ := net.SplitHostPort(target)
	if sni == "" {
		sni = host
	}
	dialer := &net.Dialer{Timeout: 5 * time.Second}
	start := time.Now()
	conn, err := tls.DialWithDialer(dialer, "tcp", target, &tls.Config{
		ServerName: sni,                        // sent as SNI AND used to verify the cert
		NextProtos: []string{"h2", "http/1.1"}, // ALPN: what we'd like to speak next
	})
	if err != nil {
		return err
	}
	defer conn.Close()
	st := conn.ConnectionState()

	fmt.Printf("%s  (TCP+TLS in %v)\n", target, time.Since(start).Round(time.Millisecond))
	fmt.Printf("  version  %s\n", tls.VersionName(st.Version))
	fmt.Printf("  cipher   %s\n", tls.CipherSuiteName(st.CipherSuite))
	fmt.Printf("  ALPN     %q\n", st.NegotiatedProtocol)
	fmt.Printf("  resumed  %v\n", st.DidResume)
	for i, c := range st.PeerCertificates {
		role := "leaf"
		if i > 0 {
			role = "intermediate"
		}
		days := int(time.Until(c.NotAfter).Hours() / 24)
		fmt.Printf("  [%d] %-12s CN=%s\n", i, role, c.Subject.CommonName)
		fmt.Printf("      issuer        %s\n", c.Issuer.CommonName)
		fmt.Printf("      valid until   %s  (%d days left)\n", c.NotAfter.Format("2006-01-02"), days)
		if i == 0 {
			fmt.Printf("      SANs          %s\n", strings.Join(c.DNSNames, ", "))
			if days < 21 {
				fmt.Println("      WARNING: renew soon")
			}
		}
	}
	fmt.Printf("  verified chain length: %d (ends at a root in YOUR trust store)\n\n", len(st.VerifiedChains[0]))
	return nil
}

// explain turns crypto/x509 errors into the fingerprints from Chapter 56.
func explain(err error) string {
	var hostErr x509.HostnameError
	var unknown x509.UnknownAuthorityError
	var invalid x509.CertificateInvalidError
	switch {
	case errors.As(err, &hostErr):
		return "NAME MISMATCH: certificate is valid for " + strings.Join(hostErr.Certificate.DNSNames, ", ")
	case errors.As(err, &unknown):
		return "UNKNOWN AUTHORITY: self-signed, private CA, or missing intermediate"
	case errors.As(err, &invalid) && invalid.Reason == x509.Expired:
		return "EXPIRED (or your clock is wrong): " + invalid.Detail
	case strings.Contains(err.Error(), "handshake failure"):
		return "HANDSHAKE REFUSED by server: often an SNI it has no certificate for (" + err.Error() + ")"
	default:
		return err.Error()
	}
}

$ go run ./tlsinspect example.com
example.com:443  (TCP+TLS in 82ms)
  version  TLS 1.3
  cipher   TLS_AES_128_GCM_SHA256
  ALPN     "h2"
  [0] leaf         CN=example.com
      issuer        Cloudflare TLS Issuing ECC CA 3
      valid until   2026-12-25  (83 days left)
      SANs          example.com, *.example.com
  [1] intermediate CN=Cloudflare TLS Issuing ECC CA 3
  ...
$ go run ./tlsinspect expired.badssl.com self-signed.badssl.com wrong.host.badssl.com
expired.badssl.com:443
  FAILED: EXPIRED (or your clock is wrong): "*.badssl.com" certificate is expired
self-signed.badssl.com:443
  FAILED: UNKNOWN AUTHORITY: self-signed, private CA, or missing intermediate
wrong.host.badssl.com:443
  FAILED: NAME MISMATCH: certificate is valid for *.badssl.com, badssl.com
$ go run ./tlsinspect -sni github.com example.com:443
  FAILED: HANDSHAKE REFUSED by server: often an SNI it has no certificate for

What to notice:

• ServerName does two jobs: it's sent as SNI (so a shared server knows which certificate to present) and it's the name the certificate is checked against. The last example asks a Cloudflare edge for a name it doesn't host, and the edge refuses the handshake before sending any certificate.
• A wildcard covers exactly one label. *.example.com matches wrong.example.com, so -sni wrong.example.com example.com passes. It does not match a.b.example.com or example.com itself, which is why the SAN list above includes both.
• ALPN "h2": the server agreed during the handshake to speak HTTP/2 next, with no extra round trip (Chapter 30).
• The verified chain ends at a root in your trust store, the operating system's on macOS and Linux. That's why "works on my laptop, fails in the container" happens: minimal container images often have no CA bundle at all.

Exercises:

1. Add an -expiry-days 30 flag that exits non-zero if any leaf expires within 30 days, and run it from cron against all your endpoints. That is a certificate-expiry monitor. Chapter 56 turns it into an operational practice.
2. Connect twice in one process with a shared tls.Config that has a ClientSessionCache (tls.NewLRUClientSessionCache(10)). Does resumed become true the second time? Not if you close straight after the handshake. In TLS 1.3 the server sends its session ticket after the handshake, so the client only receives it if it reads from the connection. Send a HEAD request and read the reply on the first connection, and the second one resumes. (Tested: resumed=false without the read, resumed=true with it.)
3. Go deeper with the Go plan: TLS inspector + cert monitor (Day 38), a full PKI generator (Day 40), and mTLS with identity extraction (Day 41).

Across the series: the HTTPS guide's TLS chapter explains the handshake from the browser's side, and the Go guide's TLS chapter covers crypto/tls server configuration.

Practice (20 min)

A. Check what a site supports:

curl -sI --http2 https://example.com | head -1        # "HTTP/2 200"
curl -sI --http1.1 https://example.com | head -1      # "HTTP/1.1 200"
curl -sI --http3 https://cloudflare.com | head -1     # needs curl built with HTTP/3

B. Look for the HTTP/3 advertisement:

curl -sI https://cloudflare.com | grep -i alt-svc
dig cloudflare.com HTTPS +short          # look for alpn="h3"

C. See QUIC traffic on your machine. You're probably already using it:

sudo tcpdump -i any -n 'udp port 443'

Browse a Google or Cloudflare-fronted site. That UDP traffic on 443 is HTTP/3.

D. Compare in your browser. DevTools → Network → right-click the column headers → enable Protocol. Reload a few sites and see h2, h3, and http/1.1 in the wild.

E. See multiplexing: in DevTools, look at the waterfall for an HTTP/2 site. Notice many requests starting at nearly the same moment, rather than in batches of six.

Practice (35 min) — the essential recipes

A. A complete connection, start to finish:

sudo tcpdump -i any -n -tttt 'host example.com' -c 30
# in another terminal:
curl -s https://example.com > /dev/null

Identify: the DNS query, the handshake, the TLS negotiation, data, and the close.

B. All connection attempts leaving your machine:

sudo tcpdump -i any -n 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'

Leave it running. You will be surprised how much your machine talks to.

C. Find who's being refused:

sudo tcpdump -i any -n 'tcp[tcpflags] & tcp-rst != 0'

D. Read HTTP in plain text:

sudo tcpdump -i any -n -A 'tcp port 80' -c 40
curl -s http://example.com > /dev/null

E. Save a capture for Wireshark:

sudo tcpdump -i any -n -s0 -w /tmp/capture.pcap 'host example.com'
# ... generate traffic ...
# Ctrl-C, then:
wireshark /tmp/capture.pcap        # or open it in the GUI

F. A rotating capture for an intermittent problem (leave it running for hours):

sudo tcpdump -i any -n -s0 -C 100 -W 10 -w /tmp/ring.pcap 'port 443'
# keeps 10 files of 100 MB each, overwriting the oldest

Practice (40 min)

A. Capture and analyse your own web browsing:

1. Start a capture in Wireshark on your main interface.
2. Load a website.
3. Stop the capture.
4. Do the five steps above in order.
5. Find the DNS query, the TCP handshake, the TLS handshake, and the data.
6. Follow → TCP Stream on the DNS query and on an HTTP request.

B. Deliberately create problems and see their fingerprints (Linux):

# 1. Packet loss
sudo tc qdisc add dev eth0 root netem loss 5%
#    ... capture a download ... then look at Expert Information
sudo tc qdisc del dev eth0 root

# 2. High latency
sudo tc qdisc add dev eth0 root netem delay 200ms
#    ... capture ... notice how slowly the window grows
sudo tc qdisc del dev eth0 root

# 3. Both, like a bad mobile connection
sudo tc qdisc add dev eth0 root netem delay 150ms 50ms loss 2%
sudo tc qdisc del dev eth0 root      # ALWAYS undo

This exercise is worth an entire textbook chapter. You are creating the exact conditions you'll later have to diagnose, and seeing what they look like.

C. Practise on sample captures. The Wireshark wiki has a large library of example captures (search "Wireshark SampleCaptures"). Open a few and try to work out what happened before reading the description.

Build it in Go (60 min) — decode a capture file yourself

Wireshark's dissectors look like magic until you write one. A .pcap file is a 24-byte header followed by records, and each record is a 16-byte header plus the raw bytes of one frame. Decoding a frame means peeling headers off in the order of Chapter 3: link layer → IP → TCP/UDP. This decoder handles Ethernet (with VLAN tags), Linux "cooked" captures from -i any (v1 and v2), macOS loopback, IPv4, IPv6, TCP, UDP, and ICMP, and it flags likely retransmissions.

// pcapread: decode a tcpdump capture file yourself -- pcap header, Ethernet,
// IPv4/IPv6, TCP, UDP -- and print a tcpdump-like line per packet, plus a
// summary that flags handshakes, resets, and likely retransmissions.
//
//	sudo tcpdump -i any -w web.pcap -c 200 'tcp port 443'   # Linux "any" = SLL headers
//	sudo tcpdump -i en0 -w web.pcap -c 200 'tcp port 443'   # Ethernet headers
//	go run ./pcapread web.pcap
package main

import (
	"bufio"
	"encoding/binary"
	"errors"
	"fmt"
	"io"
	"net/netip"
	"os"
	"strings"
	"time"
)

const (
	linkEthernet = 1   // DLT_EN10MB
	linkRaw      = 101 // DLT_RAW: starts at the IP header
	linkSLL      = 113 // DLT_LINUX_SLL: tcpdump -i any on older Linux
	linkSLL2     = 276 // DLT_LINUX_SLL2: tcpdump -i any on current Linux
	linkNull     = 0   // DLT_NULL: macOS loopback (lo0)
)

type flowKey struct {
	src, dst netip.AddrPort
}

func main() {
	if len(os.Args) != 2 {
		fmt.Println("usage: pcapread file.pcap")
		os.Exit(2)
	}
	f, err := os.Open(os.Args[1])
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	defer f.Close()
	r := bufio.NewReader(f)

	// Global header: 24 bytes. The magic number tells us byte order and
	// whether timestamps are micro- or nanoseconds.
	var gh [24]byte
	if _, err := io.ReadFull(r, gh[:]); err != nil {
		fmt.Println("not a pcap file:", err)
		os.Exit(1)
	}
	var bo binary.ByteOrder = binary.LittleEndian
	nano := false
	switch binary.LittleEndian.Uint32(gh[:]) {
	case 0xa1b2c3d4:
	case 0xa1b23c4d:
		nano = true
	case 0xd4c3b2a1:
		bo = binary.BigEndian
	case 0x4d3cb2a1:
		bo, nano = binary.BigEndian, true
	default:
		fmt.Println("unknown magic (pcapng? convert with: editcap -F pcap in.pcapng out.pcap)")
		os.Exit(1)
	}
	link := bo.Uint32(gh[20:])
	fmt.Printf("pcap v%d.%d  snaplen=%d  linktype=%d\n\n", bo.Uint16(gh[4:]), bo.Uint16(gh[6:]), bo.Uint32(gh[16:]), link)

	seen := map[flowKey]map[uint32]bool{} // per-direction sequence numbers carrying data
	var count, syns, rsts, retrans int
	var first time.Time
	for {
		var rh [16]byte
		if _, err := io.ReadFull(r, rh[:]); err != nil {
			break // io.EOF: done
		}
		sec, frac := bo.Uint32(rh[0:]), bo.Uint32(rh[4:])
		capLen, origLen := bo.Uint32(rh[8:]), bo.Uint32(rh[12:])
		data := make([]byte, capLen)
		if _, err := io.ReadFull(r, data); err != nil {
			fmt.Println("truncated record")
			break
		}
		ns := int64(frac) * 1000
		if nano {
			ns = int64(frac)
		}
		ts := time.Unix(int64(sec), ns)
		if first.IsZero() {
			first = ts
		}
		count++

		line, info, err := decode(link, data)
		if err != nil {
			fmt.Printf("%9.6f  [%d bytes] %v\n", ts.Sub(first).Seconds(), origLen, err)
			continue
		}
		if info.tcp {
			if info.flags&0x02 != 0 {
				syns++
			}
			if info.flags&0x04 != 0 {
				rsts++
			}
			if info.payload > 0 { // same direction, same seq, carrying data again = retransmission
				k := flowKey{info.src, info.dst}
				if seen[k] == nil {
					seen[k] = map[uint32]bool{}
				}
				if seen[k][info.seq] {
					retrans++
					line += "   <-- RETRANSMISSION?"
				}
				seen[k][info.seq] = true
			}
		}
		fmt.Printf("%9.6f  %s\n", ts.Sub(first).Seconds(), line)
	}
	fmt.Printf("\n%d packets, %d SYN, %d RST, %d likely retransmissions\n", count, syns, rsts, retrans)
}

type pktInfo struct {
	src, dst netip.AddrPort
	tcp      bool
	flags    byte
	seq      uint32
	payload  int
}

// decode peels the layers off one packet, outermost first.
func decode(link uint32, b []byte) (string, pktInfo, error) {
	var etherType uint16
	switch link {
	case linkEthernet: // dst MAC(6) src MAC(6) EtherType(2)
		if len(b) < 14 {
			return "", pktInfo{}, errors.New("short ethernet")
		}
		etherType, b = binary.BigEndian.Uint16(b[12:]), b[14:]
		if etherType == 0x8100 && len(b) >= 4 { // 802.1Q VLAN tag
			etherType, b = binary.BigEndian.Uint16(b[2:]), b[4:]
		}
	case linkSLL: // 16-byte "cooked" header, protocol in the last 2 bytes
		if len(b) < 16 {
			return "", pktInfo{}, errors.New("short SLL")
		}
		etherType, b = binary.BigEndian.Uint16(b[14:]), b[16:]
	case linkSLL2: // 20-byte "cooked v2" header, protocol in the FIRST 2 bytes
		if len(b) < 20 {
			return "", pktInfo{}, errors.New("short SLL2")
		}
		etherType, b = binary.BigEndian.Uint16(b[0:]), b[20:]
	case linkNull: // 4-byte host-order address family
		if len(b) < 4 {
			return "", pktInfo{}, errors.New("short null header")
		}
		etherType, b = 0x0800, b[4:]
		if b[0]>>4 == 6 {
			etherType = 0x86DD
		}
	case linkRaw:
		etherType = 0x0800
		if len(b) > 0 && b[0]>>4 == 6 {
			etherType = 0x86DD
		}
	default:
		return "", pktInfo{}, fmt.Errorf("linktype %d not supported", link)
	}

	var src, dst netip.Addr
	var proto byte
	var ttl byte
	switch etherType {
	case 0x0800: // IPv4: header length is in 32-bit words
		if len(b) < 20 {
			return "", pktInfo{}, errors.New("short IPv4")
		}
		ihl := int(b[0]&0x0F) * 4
		total := int(binary.BigEndian.Uint16(b[2:]))
		ttl, proto = b[8], b[9]
		src, dst = netip.AddrFrom4([4]byte(b[12:16])), netip.AddrFrom4([4]byte(b[16:20]))
		if total < len(b) {
			b = b[:total] // drop Ethernet padding
		}
		b = b[ihl:]
	case 0x86DD: // IPv6: fixed 40-byte header
		if len(b) < 40 {
			return "", pktInfo{}, errors.New("short IPv6")
		}
		proto, ttl = b[6], b[7]
		src, dst = netip.AddrFrom16([16]byte(b[8:24])), netip.AddrFrom16([16]byte(b[24:40]))
		b = b[40:]
	case 0x0806:
		return "ARP", pktInfo{}, nil
	default:
		return fmt.Sprintf("ethertype 0x%04x", etherType), pktInfo{}, nil
	}

	switch proto {
	case 6: // TCP
		if len(b) < 20 {
			return "", pktInfo{}, errors.New("short TCP")
		}
		sp, dp := binary.BigEndian.Uint16(b[0:]), binary.BigEndian.Uint16(b[2:])
		seq, ack := binary.BigEndian.Uint32(b[4:]), binary.BigEndian.Uint32(b[8:])
		off := int(b[12]>>4) * 4
		flags := b[13]
		win := binary.BigEndian.Uint16(b[14:])
		payload := max(0, len(b)-off)
		info := pktInfo{netip.AddrPortFrom(src, sp), netip.AddrPortFrom(dst, dp), true, flags, seq, payload}
		return fmt.Sprintf("TCP %s > %s [%s] seq %d ack %d win %d len %d ttl %d",
			info.src, info.dst, tcpFlags(flags), seq, ack, win, payload, ttl), info, nil
	case 17: // UDP
		if len(b) < 8 {
			return "", pktInfo{}, errors.New("short UDP")
		}
		sp, dp := binary.BigEndian.Uint16(b[0:]), binary.BigEndian.Uint16(b[2:])
		return fmt.Sprintf("UDP %s > %s len %d", netip.AddrPortFrom(src, sp), netip.AddrPortFrom(dst, dp), len(b)-8), pktInfo{}, nil
	case 1, 58:
		return fmt.Sprintf("ICMP %s > %s type %d", src, dst, b[0]), pktInfo{}, nil
	}
	return fmt.Sprintf("IP proto %d %s > %s", proto, src, dst), pktInfo{}, nil
}

// tcpFlags renders flags the way tcpdump does: S=SYN F=FIN R=RST P=PSH .=ACK
func tcpFlags(f byte) string {
	var s strings.Builder
	for _, x := range []struct {
		bit byte
		c   string
	}{{0x02, "S"}, {0x01, "F"}, {0x04, "R"}, {0x08, "P"}, {0x10, "."}} {
		if f&x.bit != 0 {
			s.WriteString(x.c)
		}
	}
	return s.String()
}

Capture something and decode it (this is a real capture of curl https://example.com on Linux):

$ sudo tcpdump -U -i any -w web.pcap 'tcp port 443' & curl -s -o /dev/null https://example.com; kill %1
$ go run ./pcapread web.pcap
pcap v2.4  snaplen=262144  linktype=276

 0.000000  TCP 172.17.0.2:46276 > 104.20.23.154:443 [S] seq 299092458 ack 0 win 65495 len 0 ttl 64
 0.023721  TCP 104.20.23.154:443 > 172.17.0.2:46276 [S.] seq 2742249108 ack 299092459 win 65408 len 0 ttl 63
 0.023741  TCP 172.17.0.2:46276 > 104.20.23.154:443 [.] seq 299092459 ack 2742249109 win 64 len 0 ttl 64
 0.025817  TCP 172.17.0.2:46276 > 104.20.23.154:443 [P.] seq 299092459 ack 2742249109 win 64 len 1571 ttl 64
 0.026065  TCP 104.20.23.154:443 > 172.17.0.2:46276 [.] seq 2742249109 ack 299094030 win 4083 len 0 ttl 63
...
21 packets, 2 SYN, 0 RST, 0 likely retransmissions

Read it the way this chapter taught you:

• Lines 1–3 are the three-way handshake from Chapter 21. Check the +1: the SYN-ACK acknowledges 299092459 = the client's ISN + 1.
• RTT = 23.7 ms, the gap between the SYN and the SYN-ACK. You just measured latency from a file.
• Line 4 has len 1571: the TLS ClientHello, which is bigger than you might expect because it carries key shares and extensions. Line 5's ACK number (299094030 = 299092459 + 1571) confirms the server received exactly that many bytes.
• win 64 is not 64 bytes. It's the raw header field. Both sides negotiated window scaling in the SYNs (Chapter 24), so the real window is 64 << wscale. Decoding the scale needs the SYN's TCP options, which is exercise 2.
• ttl 63 on packets from the server: it set TTL 64, and one router (here the Docker bridge's NAT) decremented it on the way.

Exercises:

1. Capture a download with loss injected (sudo tc qdisc add dev eth0 root netem loss 15%). In testing, the decoder flagged the retransmitted segment: ... [P.] seq 1861632025 ... len 9904 ttl 63 <-- RETRANSMISSION?. Check it against Wireshark's tcp.analysis.retransmission filter.
2. Parse the TCP options in SYN packets (kind 2 = MSS, 3 = window scale, 4 = SACK permitted, 8 = timestamps) and print the real window size.
3. Add a per-connection summary keyed by the 4-tuple: bytes in each direction, duration, handshake RTT. That's Wireshark's Statistics → Conversations view.
4. Pull the SNI out of the ClientHello (the first payload byte is 0x16 for a TLS handshake; the SNI is extension type 0). This is how firewalls and L4 proxies route TLS without decrypting it. The Go plan's SNI router (Day 55) builds on exactly this.

Practice (25 min)

A. Audit what's listening on your machine:

ss -tulnp            # Linux
sudo lsof -iTCP -sTCP:LISTEN -n -P   # macOS

For each entry, ask: what is this, and should it be reachable from the network? Anything on 0.0.0.0 is exposed to your local network. Do this on any server you run.

B. Watch connections appear and disappear:

watch -n1 'ss -tn state established | head -20'

Then browse the web and watch them come and go.

C. Investigate one connection in depth:

# start a big download
curl -o /dev/null https://speed.cloudflare.com/__down?bytes=100000000 &
# watch its internals
watch -n0.5 "ss -tin | grep -A2 cloudflare"

Watch cwnd grow (slow start!), rtt settle, and retrans hopefully stay at 0.

D. Find the process using a port:

sudo ss -tulnp | grep :8000       # Linux
sudo lsof -i :8000                 # macOS/Linux

This is the answer to "address already in use".

E. Check the kernel's TCP counters:

nstat -az | grep -Ei 'retrans|timeout|listendrop|listenoverflow'  # Linux
netstat -s -p tcp | head -30                                       # macOS

ListenOverflows climbing means a server's accept queue is full — a real and often-missed cause of intermittent connection failures.

Build it in Go (30 min, Linux) — ss -ti from inside your program

ss -tin reads its numbers from the kernel with getsockopt(TCP_INFO). Your own program can make the same call on its own sockets and log the kernel's view of each connection: RTT, congestion window, retransmissions. That turns "the network is slow" into numbers. This lab uploads 40 MB through Go's HTTP client and samples TCP_INFO every 250 ms.

//go:build linux

// tcpinfo: read the kernel's per-connection TCP state (the same data `ss -ti`
// shows) from inside your own program while it UPLOADS -- the sender is the
// side whose congestion window (cwnd) grows and shrinks.
//
//	go run ./tcpinfo https://speed.cloudflare.com/__up
//	# then add loss and run again:  sudo tc qdisc add dev eth0 root netem loss 2%
package main

import (
	"context"
	"fmt"
	"io"
	"net"
	"net/http"
	"os"
	"syscall"
	"time"
	"unsafe"
)

func main() {
	if len(os.Args) != 2 {
		fmt.Println("usage: tcpinfo URL")
		os.Exit(2)
	}
	conns := make(chan *net.TCPConn, 1)
	tr := &http.Transport{
		ForceAttemptHTTP2: false,
		DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
			c, err := (&net.Dialer{}).DialContext(ctx, network, addr)
			if tc, ok := c.(*net.TCPConn); ok {
				conns <- tc // keep a handle on the raw TCP socket under the HTTP client
			}
			return c, err
		},
	}
	const size = 40 << 20 // 40 MB of zeros
	done := make(chan error, 1)
	go func() {
		resp, err := (&http.Client{Transport: tr}).Post(os.Args[1], "application/octet-stream",
			io.LimitReader(zeros{}, size))
		if err == nil {
			resp.Body.Close()
		}
		done <- err
	}()
	tc := <-conns

	fmt.Println("  time   rtt(ms) rttvar  cwnd(segs)  ssthresh  unacked  lost  total_retrans")
	start := time.Now()
	tick := time.NewTicker(250 * time.Millisecond)
	defer tick.Stop()
	for {
		select {
		case err := <-done:
			fmt.Printf("done in %v (err=%v)\n", time.Since(start).Round(time.Millisecond), err)
			return
		case <-tick.C:
			ti, err := getTCPInfo(tc)
			if err != nil {
				fmt.Println("getsockopt:", err)
				return
			}
			ssthresh := fmt.Sprint(ti.Snd_ssthresh)
			if ti.Snd_ssthresh >= 1<<30 {
				ssthresh = "inf" // no loss seen yet: still in slow start
			}
			fmt.Printf("%6.2fs  %7.2f %6.2f  %10d  %8s  %7d  %4d  %13d\n",
				time.Since(start).Seconds(), float64(ti.Rtt)/1000, float64(ti.Rttvar)/1000,
				ti.Snd_cwnd, ssthresh, ti.Unacked, ti.Lost, ti.Total_retrans)
		}
	}
}

// getTCPInfo calls getsockopt(fd, IPPROTO_TCP, TCP_INFO). SyscallConn gives
// us the file descriptor without taking it away from Go's network poller.
func getTCPInfo(c *net.TCPConn) (*syscall.TCPInfo, error) {
	raw, err := c.SyscallConn()
	if err != nil {
		return nil, err
	}
	var info syscall.TCPInfo
	var sysErr error
	err = raw.Control(func(fd uintptr) {
		size := uint32(syscall.SizeofTCPInfo)
		_, _, e := syscall.Syscall6(syscall.SYS_GETSOCKOPT, fd,
			syscall.IPPROTO_TCP, syscall.TCP_INFO,
			uintptr(unsafe.Pointer(&info)), uintptr(unsafe.Pointer(&size)), 0)
		if e != 0 {
			sysErr = e
		}
	})
	if err != nil {
		return nil, err
	}
	return &info, sysErr
}

type zeros struct{}

func (zeros) Read(p []byte) (int, error) { clear(p); return len(p), nil }

$ go run ./tcpinfo https://speed.cloudflare.com/__up
  time   rtt(ms) rttvar  cwnd(segs)  ssthresh  unacked  lost  total_retrans
  ...one row every 250 ms...

$ sudo tc qdisc add dev eth0 root netem loss 2%     # then run it again

Run it on a real Linux host or VM. Inside Docker Desktop on a Mac, the connection is proxied by the VM's network layer, so RTT and cwnd describe a hop to the proxy, not the path to Cloudflare. What to look for:

• Without loss: ssthresh shows inf at first and cwnd climbs quickly. That's slow start. total_retrans stays at 0.
• With 2% loss: total_retrans starts counting, ssthresh becomes a real number the first time loss is detected, and cwnd drops to around it and then grows more slowly. You're watching congestion control react.
• rtt vs rttvar: a large variance relative to RTT means jitter. In testing, a lossy run started with rtt 814 ms and rttvar 480 ms: the first samples included retransmission delays.

What to notice:

• SyscallConn().Control() hands you the raw file descriptor without taking it away from Go's runtime. It's the safe escape hatch for socket options and getsockopt calls that the net package doesn't wrap. (Calling File() instead duplicates the descriptor and puts it into blocking mode.)
• The custom DialContext is how you get hold of the TCP socket under a high-level HTTP client. The same hook is used for custom DNS, SOCKS proxies, and the SSRF protection in the HTTPS guide.
• ssthresh = inf means no loss has happened yet, so the connection is still in slow start (Chapter 25). After a loss, ssthresh is set to roughly half of cwnd, and the congestion window grows linearly from there.
• Which side to measure: cwnd describes the sending side. On a download your client's cwnd stays at 10 because it hardly sends anything. To study a slow download, measure on the server.

Exercise: make it a library. Write func LogTCPInfo(c net.Conn, every time.Duration) and call it from a real service on its slowest outbound connection, emitting rtt, cwnd, and total_retrans as structured log fields (Go guide: log/slog). When someone next says "the database is slow", you'll know whether the network was involved.

Practice (30 min) — break things on purpose

The single best way to learn troubleshooting is to cause the failures yourself, so you recognise them later. Do these on a spare machine or VM, and undo each one.

# 1. Break DNS
sudo cp /etc/resolv.conf /tmp/resolv.backup
echo "nameserver 192.0.2.1" | sudo tee /etc/resolv.conf   # a black hole
# symptom: ping by IP works, by name doesn't
sudo cp /tmp/resolv.backup /etc/resolv.conf                # UNDO

# 2. Break the default route
ip route | grep default                                    # note it first!
sudo ip route del default
# symptom: local works, internet doesn't, "Network is unreachable"
sudo ip route add default via <your gateway>               # UNDO

# 3. Break the MTU
sudo ip link set eth0 mtu 900
# symptom: small things work, large downloads stall
sudo ip link set eth0 mtu 1500                             # UNDO

# 4. Simulate loss and latency
sudo tc qdisc add dev eth0 root netem loss 10% delay 200ms
sudo tc qdisc del dev eth0 root                            # UNDO

# 5. Block a port
sudo iptables -A OUTPUT -p tcp --dport 443 -j DROP
# symptom: TIMEOUT (not refused) -- feel the difference
sudo iptables -D OUTPUT -p tcp --dport 443 -j DROP         # UNDO

For each one: before undoing it, run the diagnostic script above and note which step first shows the failure. Write it in notes.md. That table becomes your personal troubleshooting reference.

Build it in Go (35 min) — the troubleshooting ladder as one command

The method in this chapter is "climb the layers and stop at the first one that fails". That's an algorithm, so here it is as a program. Each rung only runs if the one below it passed, and the program tells you which layer to investigate.

// netcheck: Chapter 34's troubleshooting ladder as one program. It climbs the
// stack -- route/interface, DNS, TCP, TLS, HTTP -- stops at the first rung
// that fails, and tells you which layer to look at.
//
//	go run ./netcheck https://example.com/
//	go run ./netcheck https://example.com:81/      # watch it stop at TCP
package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"net"
	"net/http"
	"net/url"
	"os"
	"time"
)

type step struct {
	layer string
	run   func() (string, error)
}

func main() {
	if len(os.Args) != 2 {
		fmt.Println("usage: netcheck URL")
		os.Exit(2)
	}
	u, err := url.Parse(os.Args[1])
	if err != nil || u.Hostname() == "" {
		fmt.Println("bad URL:", os.Args[1])
		os.Exit(2)
	}
	host, port := u.Hostname(), u.Port()
	if port == "" {
		port = map[string]string{"http": "80", "https": "443"}[u.Scheme]
	}
	var ips []net.IP
	var addr string

	steps := []step{
		{"L3 route", func() (string, error) {
			c, err := net.Dial("udp", "1.1.1.1:53") // sends nothing; just asks the routing table
			if err != nil {
				return "", fmt.Errorf("no route to the internet: %w", err)
			}
			defer c.Close()
			src := c.LocalAddr().(*net.UDPAddr).IP
			// Which interface owns that source address? That's the one that matters.
			ifs, _ := net.Interfaces()
			for _, i := range ifs {
				as, _ := i.Addrs()
				for _, a := range as {
					if n, ok := a.(*net.IPNet); ok && n.IP.Equal(src) {
						return fmt.Sprintf("default route via %s (up=%v), source %s", i.Name, i.Flags&net.FlagUp != 0, src), nil
					}
				}
			}
			return "default route exists, source " + src.String(), nil
		}},
		{"L7 DNS", func() (string, error) {
			ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
			defer cancel()
			start := time.Now()
			addrs, err := net.DefaultResolver.LookupIPAddr(ctx, host)
			if err != nil {
				return "", err
			}
			for _, a := range addrs {
				ips = append(ips, a.IP)
			}
			return fmt.Sprintf("%s -> %v in %v", host, ips, time.Since(start).Round(time.Millisecond)), nil
		}},
		{"L4 TCP", func() (string, error) {
			addr = net.JoinHostPort(ips[0].String(), port)
			start := time.Now()
			c, err := net.DialTimeout("tcp", addr, 3*time.Second)
			if err != nil {
				return "", err
			}
			c.Close()
			return fmt.Sprintf("connected to %s in %v (~1 RTT)", addr, time.Since(start).Round(time.Millisecond)), nil
		}},
		{"L5/6 TLS", func() (string, error) {
			if u.Scheme != "https" {
				return "skipped (plain http)", nil
			}
			start := time.Now()
			c, err := tls.DialWithDialer(&net.Dialer{Timeout: 3 * time.Second}, "tcp", addr,
				&tls.Config{ServerName: host})
			if err != nil {
				return "", err
			}
			defer c.Close()
			st := c.ConnectionState()
			days := int(time.Until(st.PeerCertificates[0].NotAfter).Hours() / 24)
			return fmt.Sprintf("%s, cert valid %d more days, TCP+TLS %v",
				tls.VersionName(st.Version), days, time.Since(start).Round(time.Millisecond)), nil
		}},
		{"L7 HTTP", func() (string, error) {
			client := &http.Client{Timeout: 5 * time.Second,
				CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
			start := time.Now()
			resp, err := client.Get(u.String())
			if err != nil {
				return "", err
			}
			resp.Body.Close()
			if resp.StatusCode >= 500 {
				return "", fmt.Errorf("server error %s", resp.Status)
			}
			return fmt.Sprintf("%s in %v (server: %q)", resp.Status, time.Since(start).Round(time.Millisecond), resp.Header.Get("Server")), nil
		}},
	}

	for _, s := range steps {
		detail, err := s.run()
		if err != nil {
			fmt.Printf("FAIL  %-11s %v\n\n=> Everything below %s works. Investigate %s first.\n", s.layer, err, s.layer, s.layer)
			os.Exit(1)
		}
		fmt.Printf("ok    %-11s %s\n", s.layer, detail)
	}
	fmt.Println("\n=> every layer answered. If users still complain, look at latency, not reachability.")
}

$ go run ./netcheck https://example.com/
ok    L3 route    default route via en0 (up=true), source 192.168.1.2
ok    L7 DNS      example.com -> [104.20.23.154 172.66.147.243 2606:4700:...] in 2ms
ok    L4 TCP      connected to 104.20.23.154:443 in 38ms (~1 RTT)
ok    L5/6 TLS    TLS 1.3, cert valid 83 more days, TCP+TLS 96ms
ok    L7 HTTP     200 OK in 514ms (server: "cloudflare")

$ go run ./netcheck https://example.com:81/
ok    L3 route    default route via en0 (up=true), source 192.168.1.2
ok    L7 DNS      example.com -> [...] in 3ms
FAIL  L4 TCP      dial tcp 172.66.147.243:81: i/o timeout

=> Everything below L4 TCP works. Investigate L4 TCP first.

What to notice:

• The order isn't the OSI order. DNS (an L7 protocol) runs before TCP, because you need the address before you can connect. The OSI guide's "layers are not steps in time" section explains why this is correct.
• Each rung is the smallest test that can prove its layer. TCP is checked with a bare connect, not an HTTP request. A failing HTTP request alone can't tell you whether DNS, TCP, TLS, or the app is at fault.
• Timing on every rung. "Everything works, but TCP took 900 ms" is also a finding: SYN loss and a retry (Chapter 21).

Exercises:

1. Break each layer in turn, using the "break things on purpose" practice above (a bad resolver, no default route, a blocked port), and check that the tool names the right layer each time.
2. Add an MTU rung: download something larger than 1,500 bytes and fail if the small request works but the large one stalls (Chapter 15's fingerprint).
3. Add -json output and run it from cron on three networks (home, office, cloud VM). You now have a minimal synthetic monitor. Chapter 46's prober lab turns that into proper metrics.

Across the series: the same ladder appears as curl -w timings in the OSI guide's debugging playbook and in the HTTPS guide's debugging chapter. The Linux guide's Appendix C is the host-side equivalent for "the server is slow".

Practice

Take the five deliberate breakages from Chapter 34, and for each one write down, in notes.md:

1. The exact symptom a user would report
2. The first command you'd run
3. What that command showed

That document is your troubleshooting playbook, and it's worth more than any chapter of this guide because you wrote it.

Practice (15 min)

# See a CDN in action -- ask different resolvers, get different answers
dig @1.1.1.1 www.cloudflare.com +short
dig @8.8.8.8 www.cloudflare.com +short

# See which anycast location answers YOU
dig CHAOS TXT id.server @1.1.1.1 +short        # Cloudflare tells you the city!

# Compare latency to a CDN vs a single-location server
ping -c 5 1.1.1.1
ping -c 5 <a server you know is far away>

# Look at the headers a CDN adds
curl -sI https://www.cloudflare.com | grep -iE 'server|cf-ray|age|cache'

Practice (20 min)

# If you have Docker: look at the software network it built
ip -br addr | grep docker              # the bridge (a virtual switch)
ip route | grep docker                 # the route to container-land
sudo iptables -t nat -L -n | head -30  # the NAT rules Docker added

# Run two containers and watch them talk
docker run -d --name a alpine sleep 600
docker run -d --name b alpine sleep 600
docker exec a ip -br addr              # container a's private IP
docker exec a ping -c2 <b's ip>

# Check a container's MTU -- often NOT 1500
docker exec a cat /sys/class/net/eth0/mtu

Practice (20 min)

# See plaintext traffic (why TLS exists)
sudo tcpdump -i any -A 'tcp port 80' -c 30 &
curl -s http://example.com > /dev/null
# your request and the response are fully readable

# Compare with HTTPS
sudo tcpdump -i any -A 'tcp port 443 and host example.com' -c 30 &
curl -s https://example.com > /dev/null
# handshake visible; content is gibberish

# Audit your own machine
ss -tulnp

Practice (20 min)

A. Watch chunked encoding happen:

curl -sv --raw http://httpbin.org/stream/3 2>&1 | tail -30

The --raw flag stops curl from decoding chunks, so you see the hex sizes and the terminating 0.

B. Compare with a fixed-length response:

curl -sI https://example.com | grep -iE 'content-length|transfer-encoding'

C. Watch a streamed response arrive over time (chunked in action):

curl -sN https://httpbin.org/drip?duration=5\&numbytes=10

Bytes trickle in over 5 seconds rather than arriving at once.

D. See 100 Continue:

head -c 1000000 /dev/urandom > /tmp/big
curl -sv -H 'Expect: 100-continue' --data-binary @/tmp/big \
     https://httpbin.org/post 2>&1 | grep '< HTTP'

You should see HTTP/1.1 100 Continue and then HTTP/1.1 200 OK.

Practice (30 min)

A. Look at real caching headers:

curl -sI https://example.com | grep -iE 'cache-control|etag|last-modified|age|vary|expires'

Try a few sites and compare. Look at a big site's static assets versus its HTML.

B. Do a conditional request yourself and get a 304:

URL=https://example.com
ET=$(curl -sI $URL | awk -F': ' 'tolower($1)=="etag"{print $2}' | tr -d '\r')
echo "ETag is: $ET"
curl -sI -H "If-None-Match: $ET" $URL | head -1      # expect: 304 Not Modified

C. See a CDN hit vs miss:

curl -sI https://www.cloudflare.com | grep -iE 'cf-cache-status|age'
curl -sI https://www.cloudflare.com | grep -iE 'cf-cache-status|age'   # run twice

Age: tells you how many seconds the cache has held it.

D. In your browser: DevTools → Network. Reload a page and look at the Size column — entries saying "(disk cache)" or "(memory cache)" cost zero network. Then tick Disable cache and reload; watch the load time and transfer size change. That difference is what caching is worth.

Practice (20 min)

A. See cookies being set:

curl -sI https://github.com | grep -i set-cookie
curl -sI https://example.com | grep -i set-cookie

B. Inspect them properly in the browser: DevTools → Application → Cookies. The columns are literally the attributes above. Look at a site you're logged into: is the session cookie Secure? HttpOnly? What's its SameSite?

C. Watch a cookie round-trip:

curl -s -c /tmp/jar.txt https://httpbin.org/cookies/set?flavour=chocolate > /dev/null
cat /tmp/jar.txt                                        # the stored cookie
curl -s -b /tmp/jar.txt https://httpbin.org/cookies     # sent back automatically

D. Measure the cost. Cookies are sent on every request to that scope — including images and API calls. Check the size:

curl -sI https://<a site you use> | grep -i set-cookie | wc -c

Multiply by every request on a page load. Large cookies are a real performance problem, which is why static assets are often served from a cookie-less domain.

Practice (25 min)

A. Simulate a preflight from the command line:

curl -sI -X OPTIONS https://httpbin.org/anything \
  -H 'Origin: https://app.example.com' \
  -H 'Access-Control-Request-Method: PUT' \
  -H 'Access-Control-Request-Headers: content-type' \
  | grep -iE 'access-control|^HTTP'

B. A simple request:

curl -sI https://httpbin.org/get -H 'Origin: https://app.example.com' \
  | grep -i access-control

C. See it fail and succeed in a browser. Open DevTools console on any site and run:

// this will be BLOCKED by CORS (no permission headers from example.com)
fetch('https://example.com').then(r => r.text()).then(console.log).catch(console.error)

// this will WORK (httpbin sends Access-Control-Allow-Origin: *)
fetch('https://httpbin.org/get').then(r => r.json()).then(console.log)

Read the error message in the console carefully — it tells you exactly which header was missing.

D. Prove that CORS is browser-only: the same https://example.com request that the browser blocked works perfectly from curl. The data was never protected; only the browser's JavaScript was restricted.

Practice (25 min)

A. Watch a WebSocket handshake:

curl -sv -N \
  -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
  -H 'Sec-WebSocket-Version: 13' \
  -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
  https://echo.websocket.org/ 2>&1 | grep -iE '^< HTTP|upgrade|sec-websocket'

Look for 101 Switching Protocols.

B. Actually use a WebSocket:

# install: apt install websocat  /  brew install websocat
websocat wss://echo.websocket.org
# type anything; it echoes back. Ctrl-C to quit.

C. Consume a real SSE stream (Wikipedia publishes every edit, live):

curl -N -H 'Accept: text/event-stream' \
  https://stream.wikimedia.org/v2/stream/recentchange | head -20

Note that it never finishes — that's the point. Ctrl-C to stop.

D. In the browser (DevTools console on any page):

const es = new EventSource('https://stream.wikimedia.org/v2/stream/recentchange');
es.onmessage = e => console.log(JSON.parse(e.data).title);
setTimeout(() => es.close(), 10000);   // stop after 10 seconds

E. See it in DevTools: Network tab → filter by WS to inspect WebSocket frames, or click an eventsource request to see the message stream.

Practice (20 min)

# compression -- see it applied, and check Vary
curl -sI -H 'Accept-Encoding: gzip, br' https://example.com \
  | grep -iE 'content-encoding|vary|content-length'

# compare sizes
curl -s https://example.com | wc -c
curl -s -H 'Accept-Encoding: gzip' https://example.com --output - | wc -c

# range requests
curl -sI https://example.com | grep -i accept-ranges
curl -s -r 0-99 https://example.com | wc -c            # expect ~100 bytes
curl -sI -H 'Range: bytes=0-99' https://example.com | grep -iE '^HTTP|content-range'

# what does your environment think about proxies?
env | grep -i proxy

# which CA bundle is each tool using?
curl -v https://example.com 2>&1 | grep -i 'CAfile'
python3 -c "import certifi; print(certifi.where())"

# are you behind TLS interception? check who signed the certificate
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -issuer

If that last command names your employer instead of a public CA, your traffic is being intercepted.