Chapter 78 -- File descriptors and the netpoller: holding thousands of connections

Build it in Go (15 min) -- hit EMFILE on purpose

//go:build linux

// fdlimit: every socket and file is a file descriptor, and the per-process
// limit (RLIMIT_NOFILE, `ulimit -n`) caps how many connections a server can
// hold. Go quietly raises the soft limit to the hard limit at startup.
//
//	docker run --rm --ulimit nofile=256:2048 -v "$PWD/bin":/b debian:bookworm /b/fdlimit
package main

import (
	"errors"
	"fmt"
	"os"
	"syscall"
)

func main() {
	var rl syscall.Rlimit
	syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rl)
	fmt.Printf("RLIMIT_NOFILE as Go sees it: soft=%d hard=%d\n", rl.Cur, rl.Max)
	fmt.Println("(Go raised the soft limit to hard-1 at startup; programs it execs get the original back)")

	var files []*os.File
	for {
		f, err := os.Open("/dev/null")
		if err != nil {
			fmt.Printf("open #%d failed: %v\n", len(files)+1, err)
			fmt.Println("EMFILE?", errors.Is(err, syscall.EMFILE), "-- 'too many open files'")
			break
		}
		files = append(files, f)
	}
	files[0].Close() // reading /proc/self/fd needs one free descriptor itself
	fds, _ := os.ReadDir("/proc/self/fd")
	fmt.Printf("descriptors open: %d (%d files + stdin/out/err + the runtime's epoll and wakeup fds)\n", len(fds), len(files)-1)
}

$ docker run --rm --ulimit nofile=256:2048 -v "$PWD/bin":/b debian:bookworm sh -c 'ulimit -Sn; /b/fdlimit'
256                                                         <- the shell's soft limit
RLIMIT_NOFILE as Go sees it: soft=2047 hard=2048            <- Go raised it: hard - 1
(Go raised the soft limit to hard-1 at startup; programs it execs get the original back)
open #2042 failed: open /dev/null: too many open files
EMFILE? true -- 'too many open files'
descriptors open: 2047 (2040 files + stdin/out/err + the runtime's epoll and wakeup fds)

What to notice:

• The shell said 256, and Go got 2,047. If you size a Go service by ulimit -n in the shell, you'll be wrong. Read /proc/<pid>/limits for the real value.
• The ReadDir of /proc/self/fd needs a descriptor too. The first draft of this lab printed "0 open" because every descriptor was in use, so even the diagnostic failed. In production, logging libraries that open files hit the same wall during an EMFILE incident. Keep a little headroom.
• Four descriptors belong to the runtime (epoll and its wakeup pipe/eventfd).

Exercises:

1. Replace os.Open("/dev/null") with net.Dial to a local listener and count how many connections fit. Then set LimitNOFILE= in a systemd unit (Chapter 82) and check /proc/<pid>/limits.
2. Add a leak: open a file per HTTP request and "forget" to close it. Watch ls /proc/<pid>/fd | wc -l climb under load, then find the leak with lsof -p <pid>.
3. Use ss -s while running the TCP/IP guide's closewait lab: each CLOSE_WAIT socket is a leaked descriptor.