Practice (30 min): build a container network by hand

Run this only on a Linux lab machine or VM. You're building what Docker's default bridge network builds:

# 1. A namespace ("the container") and a virtual cable
sudo ip netns add app
sudo ip link add veth-host type veth peer name veth-app
sudo ip link set veth-app netns app

# 2. A bridge ("docker0") on the host, with the host end plugged in
sudo ip link add br0 type bridge
sudo ip addr add 10.200.0.1/24 dev br0
sudo ip link set br0 up
sudo ip link set veth-host master br0
sudo ip link set veth-host up

# 3. Inside the namespace: an address, loopback, and a default route
sudo ip netns exec app ip addr add 10.200.0.2/24 dev veth-app
sudo ip netns exec app ip link set lo up
sudo ip netns exec app ip link set veth-app up
sudo ip netns exec app ip route add default via 10.200.0.1

ping -c 2 10.200.0.2                          # host -> "container" works

# 4. Let the namespace reach the internet: forwarding + NAT (MASQUERADE)
sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -t nat -A POSTROUTING -s 10.200.0.0/24 ! -o br0 -j MASQUERADE
sudo ip netns exec app ping -c 2 1.1.1.1      # works only after step 4

# 5. Watch conntrack record the NAT translation
sudo conntrack -L -p icmp 2>/dev/null | head

Clean up:

sudo ip netns del app
sudo ip link del br0
sudo iptables -t nat -D POSTROUTING -s 10.200.0.0/24 ! -o br0 -j MASQUERADE

Step 5 prints a line like this (from a real run):

icmp 1 29 src=10.200.0.2 dst=1.1.1.1 type=8 ... src=1.1.1.1 dst=172.17.0.2 type=0 ...

Read it as two halves. The original direction (src=10.200.0.2 dst=1.1.1.1) is what the namespace sent. The reply direction (src=1.1.1.1 dst=172.17.0.2) is what conntrack expects back: replies come to the host's address, and conntrack rewrites them back to 10.200.0.2. That one table entry is the whole of NAT.

Step 4 is the one people forget. Without IP forwarding the host won't route the namespace's packets, and without MASQUERADE the replies have nowhere to go: 10.200.0.2 is a private address the internet can't route back to (Chapter 12).

Build it in Go (20 min) — overflow an accept queue on purpose

The accept queue is the least understood queue in this list. The kernel completes handshakes without your program's involvement and parks finished connections until the program calls Accept(). A program that stops accepting (deadlocked, all workers busy, a long GC pause in older runtimes) keeps "accepting" connections until the queue is full. After that, new SYNs are silently dropped and clients see 1-second, 3-second connect delays while your dashboards show an idle process.

// acceptq: fill a listening socket's accept queue on purpose.
//
// The kernel completes TCP handshakes on its own and parks finished
// connections in the accept queue until the program calls Accept(). A program
// that stops calling Accept() (deadlocked, GC-paused, all workers busy) still
// "accepts" connections -- until the queue is full. Then new SYNs are dropped
// and clients see slow connects (SYN retries at 1 s, 3 s, ...).
//
//	go run ./acceptq
//	# Linux, in another terminal:  ss -ltn 'sport = :7400'    (Recv-Q = queued, Send-Q = queue size)
//	#                               nstat -az TcpExtListenOverflows TcpExtListenDrops
package main

import (
	"fmt"
	"net"
	"time"
)

func main() {
	ln, err := net.Listen("tcp", "127.0.0.1:7400") // Go asks for backlog = net.core.somaxconn
	if err != nil {
		panic(err)
	}
	defer ln.Close()
	fmt.Println("listening, but NEVER calling Accept()")

	var conns []net.Conn
	for i := 1; i <= 5000; i++ {
		start := time.Now()
		c, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
		took := time.Since(start)
		if err != nil {
			fmt.Printf("connection %d: %v after %v  <- queue full, SYN dropped\n", i, err, took.Round(time.Millisecond))
			break
		}
		conns = append(conns, c)
		if took > 500*time.Millisecond {
			fmt.Printf("connection %d took %v  <- SYN was dropped and retried\n", i, took.Round(time.Millisecond))
		}
		if i%1000 == 0 {
			fmt.Printf("%d connections completed the handshake without Accept()\n", i)
		}
	}
	fmt.Printf("\n%d connections are waiting in the accept queue.\n", len(conns))
	fmt.Println("Look now with ss/netstat; exiting in 30s.")
	time.Sleep(30 * time.Second)
}

Real output on Linux:

$ sysctl net.core.somaxconn
net.core.somaxconn = 4096
$ go run ./acceptq
listening, but NEVER calling Accept()
1000 connections completed the handshake without Accept()
...
4000 connections completed the handshake without Accept()
connection 4098: dial tcp 127.0.0.1:7400: i/o timeout after 2.001s  <- queue full, SYN dropped

$ ss -ltn 'sport = :7400'
State  Recv-Q Send-Q Local Address:Port
LISTEN 4097   4096       127.0.0.1:7400       <- 4097 waiting, queue size 4096
$ nstat -az TcpExtListenOverflows TcpExtListenDrops
TcpExtListenOverflows           2
TcpExtListenDrops               2

What to notice: for a LISTEN socket, ss reuses its columns: Recv-Q is the current accept-queue length and Send-Q is its maximum. Go asks for a backlog of net.core.somaxconn, so the queue held 4,096 (plus one, a long-standing Linux off-by-one) before dropping. ListenOverflows is the counter to alert on. It means "my process wasn't calling accept() fast enough".

Exercises:

1. Add a goroutine that calls Accept() once every 10 ms. What's the steady-state queue length now? (Little's law: arrival rate × time in queue.)
2. Lower somaxconn to 128 (sudo sysctl -w net.core.somaxconn=128), rerun, then restore it. Older kernels defaulted to 128, which is why "raise somaxconn" appears in every old tuning guide.
3. Read how containers get their own copies of these sysctls in the Linux guide's container chapter. somaxconn is per network namespace.