Chapter 23 — TCP part 3: closing a connection, and the states

Practice (25 min)

A. Watch a full close:

sudo tcpdump -i any -n 'tcp port 443 and host example.com'
curl -s https://example.com > /dev/null

At the end of the capture, find the [F.] (FIN+ACK) packets and the final [.].

B. Watch a reset:

sudo tcpdump -i any -n 'tcp port 9999' &
nc -v 127.0.0.1 9999          # nothing is listening

You'll see [R.] — an RST.

C. See the state histogram on your own machine:

ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

Then open a lot of browser tabs and run it again.

D. Create TIME_WAIT deliberately:

for i in $(seq 1 20); do curl -s -o /dev/null http://example.com; done
ss -tan state time-wait | wc -l
# wait 60 seconds, run again -- they disappear

E. Find any CLOSE_WAIT on your system (and identify the guilty program):

ss -tanp state close-wait          # Linux
lsof -i -sTCP:CLOSE_WAIT           # macOS/Linux

If you find some that persist, you've found a real bug in some software.

Build it in Go (30 min) — manufacture a CLOSE_WAIT leak, then fix it

TIME_WAIT is normal. CLOSE_WAIT that keeps growing is always a bug in your program: the peer has closed, the kernel has acknowledged its FIN, and the socket is waiting for your code to call close(). It will wait forever. Each leaked socket holds a file descriptor, and eventually the process hits its open-files limit and stops accepting anything (too many open files).

// closewait: manufacture the CLOSE_WAIT leak from Chapter 23, watch it with
// ss/lsof, then fix it -- and see a clean half-close with CloseWrite.
//
//	go run ./closewait -leak     # server forgets to Close
//	go run ./closewait           # server closes properly
package main

import (
	"bufio"
	"flag"
	"fmt"
	"io"
	"net"
	"os"
	"time"
)

func main() {
	leak := flag.Bool("leak", false, "simulate the bug: never close accepted connections")
	flag.Parse()

	ln, err := net.Listen("tcp", "127.0.0.1:7070")
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	fmt.Printf("server pid %d on %s\n", os.Getpid(), ln.Addr())
	fmt.Println("in another terminal run:")
	fmt.Println("  Linux:  watch -n1 \"ss -tan state close-wait '( sport = :7070 )'\"")
	fmt.Println("  macOS:  watch -n1 \"netstat -an -p tcp | grep 7070\"")

	go func() {
		for {
			c, err := ln.Accept()
			if err != nil {
				return
			}
			go handle(c, *leak)
		}
	}()

	// Ten clients connect, send a request, half-close, read the reply, leave.
	for i := 0; i < 10; i++ {
		c, err := net.Dial("tcp", ln.Addr().String())
		if err != nil {
			fmt.Println(err)
			continue
		}
		fmt.Fprintf(c, "request %d\n", i)
		c.(*net.TCPConn).CloseWrite()                   // send FIN: "I'm done talking, but still listening"
		reply, _ := bufio.NewReader(c).ReadString('\n') // the server can still answer after our FIN
		fmt.Printf("client %d got %q\n", i, reply)
		c.Close()
	}
	fmt.Println("clients finished. Server stays up 60s so you can look; Ctrl-C to quit.")
	time.Sleep(60 * time.Second)
}

func handle(c net.Conn, leak bool) {
	req, _ := io.ReadAll(c) // returns when the client's FIN arrives (EOF)
	fmt.Fprintf(c, "ok: %s", req)
	if leak {
		// BUG: no c.Close(). The client's FIN was acknowledged, so the kernel
		// moves our side to CLOSE_WAIT and waits for US to close. It will wait
		// forever, holding a file descriptor and kernel memory.
		return
	}
	c.Close()
}

Real output on Linux:

$ go run ./closewait -leak &
$ ss -tan state close-wait '( sport = :7070 )'
Recv-Q Send-Q Local Address:Port Peer Address:Port
0      0          127.0.0.1:7070    127.0.0.1:55228
0      0          127.0.0.1:7070    127.0.0.1:55258
...                                                  <- 10 sockets, and they never go away

$ go run ./closewait            # the fixed version
$ ss -tan state close-wait '( sport = :7070 )' | tail -n +2 | wc -l
0

What to notice:

• The client side is in FIN_WAIT_2 (run netstat -an | grep 7070 and you'll see both halves). The client sent its FIN and got an ACK, and is now waiting for the server's FIN, which never comes.
• CloseWrite() is a half-close: it sends FIN ("I've finished sending") but keeps reading. The server's io.ReadAll returns when that FIN arrives, and the server can still reply. HTTP/1.0 used exactly this pattern, and so do nc -N and many proxies.
• In Go, the fix is almost always a missing defer c.Close(), often on an error path, or a resp.Body from net/http that was never closed. That last one leaks the same way on the client side.

Exercises:

1. While -leak runs, check the file descriptors: ls /proc/$(pgrep closewait)/fd | wc -l. Then lower the limit with ulimit -n 64 and change the loop to 100 clients. What error do you get? (Linux guide, Chapter 43 covers ulimit.)
2. Write the same bug with net/http: a client that does resp, _ := http.Get(url) and never closes resp.Body. Watch the connections pile up in ESTABLISHED instead. Which side is now leaking?
3. The Go plan's Day 52 exercise has a production-style version with a leak detector. Do it next.