Chapter 74 -- Processes from Go: exec, exit codes, signals, and PID 1

Build it in Go (20 min) -- make zombies, then stop making them

//go:build linux

// pid1: the job nobody tells you your program has when it is PID 1 in a
// container -- reaping orphaned processes. Run it as a container's entrypoint.
//
//	docker run --rm -v "$PWD/bin":/b debian:bookworm /b/pid1           # zombies pile up
//	docker run --rm -v "$PWD/bin":/b debian:bookworm /b/pid1 -reap     # we reap them
//	docker run --rm --init -v "$PWD/bin":/b debian:bookworm /b/pid1    # tini reaps them
package main

import (
	"flag"
	"fmt"
	"os"
	"os/exec"
	"os/signal"
	"path/filepath"
	"strings"
	"syscall"
	"time"
)

func main() {
	reap := flag.Bool("reap", false, "reap orphaned children (what an init process must do)")
	flag.Parse()
	fmt.Println("my PID is", os.Getpid())

	if *reap {
		go reaper()
	}
	for i := 0; i < 5; i++ {
		// sh starts a background sleep and exits at once. The sleep is now an
		// ORPHAN, and the kernel re-parents orphans to PID 1: us.
		cmd := exec.Command("sh", "-c", "sleep 0.2 & exit 0")
		cmd.Run() // we reap sh itself; nobody has waited for the sleep yet
	}
	time.Sleep(time.Second) // the orphaned sleeps exit in the meantime
	fmt.Println("zombie processes now:", zombies())
}

// reaper waits for ANY child whenever SIGCHLD arrives. In a real init this
// loop must not race with exec.Cmd.Wait for children you started yourself --
// one reason to use a tiny dedicated init (tini, dumb-init, docker --init).
func reaper() {
	ch := make(chan os.Signal, 16)
	signal.Notify(ch, syscall.SIGCHLD)
	for range ch {
		for {
			var ws syscall.WaitStatus
			pid, err := syscall.Wait4(-1, &ws, syscall.WNOHANG, nil)
			if pid <= 0 || err != nil {
				break
			}
		}
	}
}

// zombies scans /proc for processes in state Z (exited, never waited for).
func zombies() int {
	n := 0
	stats, _ := filepath.Glob("/proc/[0-9]*/stat")
	for _, f := range stats {
		b, err := os.ReadFile(f)
		if err != nil {
			continue
		}
		// format: pid (comm) STATE ...  -- comm may contain spaces, so split after ')'
		if i := strings.LastIndex(string(b), ")"); i > 0 && strings.HasPrefix(string(b[i+2:]), "Z") {
			n++
		}
	}
	return n
}

Run it three ways (real output):

$ docker run --rm -v "$PWD/bin":/b debian:bookworm /b/pid1
my PID is 1
zombie processes now: 5

$ docker run --rm -v "$PWD/bin":/b debian:bookworm /b/pid1 -reap
my PID is 1
zombie processes now: 0

$ docker run --rm --init -v "$PWD/bin":/b debian:bookworm /b/pid1
my PID is 7
zombie processes now: 0

What to notice:

• Five zombies from five innocent shell commands. Each sh -c "sleep 0.2 &" left an orphaned sleep that became ours. A service that runs scripts thousands of times a day in a container eventually runs out of PIDs.
• docker --init puts tini in as PID 1. It reaps zombies and forwards signals, and our program became PID 7. In Kubernetes, use an init binary in the image (tini, dumb-init), or share the pod's process namespace with shareProcessNamespace: true so the pause container reaps.
• The -reap loop races with exec.Cmd.Wait: it can collect a child that cmd.Run was waiting for, so Run returns "no child processes". That race is the main reason to use a dedicated init rather than reaping inside your application.

Exercises:

1. Run /b/pid1 with docker run and, from another terminal, docker exec <id> ps -eo pid,ppid,stat,comm. Find the Z entries and their parent.
2. Remove the signal.Notify from Chapter 19 of the HTTPS guide's server and run it as PID 1. How long does docker stop take now? (10 s, Docker's default grace period, then SIGKILL.)
3. Write runWithTimeout(cmd string, d time.Duration) that kills the whole process group on timeout, and test it with sh -c "sleep 100 & sleep 100".