Chapter 75 -- CPU limits: GOMAXPROCS, cgroups, and throttling

Build it in Go (15 min) -- measure throttling

// gomaxprocs: what does a Go program think it can use, versus what the
// container's cgroup actually allows?
//
//	go run ./gomaxprocs
//	docker run --rm --cpus=1.5 -v "$PWD":/w -w /w golang:1.26 go run ./gomaxprocs
package main

import (
	"fmt"
	"os"
	"runtime"
	"strings"
	"sync"
	"time"
)

func main() {
	fmt.Println("runtime.NumCPU()   =", runtime.NumCPU(), " (CPUs the OS lets this process run on)")
	fmt.Println("GOMAXPROCS         =", runtime.GOMAXPROCS(0), " (OS threads running Go code at once)")
	if b, err := os.ReadFile("/sys/fs/cgroup/cpu.max"); err == nil {
		f := strings.Fields(string(b)) // "<quota> <period>" or "max <period>"
		fmt.Printf("cgroup cpu.max     = %s", b)
		if f[0] != "max" {
			fmt.Printf("                   = %s / %s µs per period -> the container's CPU limit\n", f[0], f[1])
		}
	} else {
		fmt.Println("cgroup cpu.max     = (not in a cgroup v2 container)")
	}

	// Burn CPU on every P for 2 seconds and see how much we got.
	before := throttled()
	var wg sync.WaitGroup
	stop := time.Now().Add(2 * time.Second)
	for i := 0; i < runtime.GOMAXPROCS(0); i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for time.Now().Before(stop) {
			}
		}()
	}
	wg.Wait()
	after := throttled()
	if before != nil && after != nil {
		fmt.Printf("during 2s of busy work: throttled %d times, for %.0f ms in total\n",
			after[0]-before[0], float64(after[1]-before[1])/1000)
	}
}

// throttled reads nr_throttled and throttled_usec from cgroup v2's cpu.stat.
func throttled() []int64 {
	b, err := os.ReadFile("/sys/fs/cgroup/cpu.stat")
	if err != nil {
		return nil
	}
	var n, us int64
	for _, line := range strings.Split(string(b), "\n") {
		f := strings.Fields(line)
		if len(f) == 2 && f[0] == "nr_throttled" {
			fmt.Sscan(f[1], &n)
		}
		if len(f) == 2 && f[0] == "throttled_usec" {
			fmt.Sscan(f[1], &us)
		}
	}
	return []int64{n, us}
}

Real output (Go 1.26, host with 14 CPUs):

$ docker run --rm -v "$PWD/bin":/b debian:bookworm /b/gomaxprocs                   # no limit
runtime.NumCPU()   = 14
GOMAXPROCS         = 14
cgroup cpu.max     = max 100000
during 2s of busy work: throttled 0 times, for 0 ms in total

$ docker run --rm --cpus=1.5 -v "$PWD/bin":/b debian:bookworm /b/gomaxprocs
runtime.NumCPU()   = 14  (CPUs the OS lets this process run on)
GOMAXPROCS         = 2   (OS threads running Go code at once)
cgroup cpu.max     = 150000 100000
during 2s of busy work: throttled 20 times, for 1011 ms in total

$ docker run --rm --cpus=1.5 -e GOMAXPROCS=8 -v "$PWD/bin":/b debian:bookworm /b/gomaxprocs
GOMAXPROCS         = 8
during 2s of busy work: throttled 21 times, for 13658 ms in total

Read it:

• Go picked GOMAXPROCS = 2 for a 1.5-CPU limit (rounded up), while NumCPU() still said 14.
• throttled_usec is summed across threads. With 8 busy threads, 13.7 s of thread-time was spent frozen during a 2 s run. Each thread was stopped for most of every period. Requests arriving then wait up to 80 ms for no reason visible in the application.
• Some throttling remains at GOMAXPROCS = 2, because 2 threads on a 1.5-CPU quota still overrun it slightly. For latency-critical services, some teams set CPU requests and no CPU limit, letting the pod use idle CPU without quota stalls. Measure before choosing.

Exercises:

1. Watch throttling on a real Kubernetes pod: kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat and compare nr_throttled before and after a load test.
2. Run with --cpus=0.5. What does Go choose for GOMAXPROCS, and how much throttling do you see? (Go never chooses fewer than 2, so a half-CPU container is throttled heavily by design. Weigh that against setting GOMAXPROCS=1 explicitly.)
3. Build the same program with Go 1.24 (docker run golang:1.24 ...) and compare GOMAXPROCS under --cpus=1.5.