Chapter 76 -- Memory limits: the Go heap, GOMEMLIMIT, and the OOM killer

In one sentence

By default Go lets its heap grow to about twice the live data before collecting, so in a container a program whose working set fits comfortably can still be OOM-killed. GOMEMLIMIT tells the garbage collector about the limit.

Why this matters

"OOMKilled, exit code 137" (Chapter 13) is one of the most common container failures, and for garbage-collected languages it often isn't a leak. The GC was pacing itself as if memory were unlimited. Knowing the two knobs (GOGC and GOMEMLIMIT) turns a mysterious crash into a configuration decision.

How it actually works

GOGC=100 (the default): run the next GC when the heap has grown 100% beyond the live heap after the last GC. Live 160 MB means the next GC at about 320 MB.
GOMEMLIMIT (Go 1.19+): a soft limit on the runtime's total memory. As memory approaches it, the GC runs more often, whatever GOGC says.
The cgroup limit (memory.max): a hard limit enforced by the kernel. On reaching it, the kernel reclaims what it can and then OOM-kills.

Set GOMEMLIMIT to roughly 90% of the container's memory limit, leaving room for non-heap memory: goroutine stacks, runtime metadata, and C libraries in cgo programs.

Build it in Go (20 min) -- an OOM kill and its cure

The program keeps 160 MB live (a "cache") and allocates 4 MB of garbage in a tight loop, like a busy service. It's built on the host and run in a 300 MB container. (Building inside that container fails: the Go compiler itself was OOM-killed when we tried, which is another lesson.)

// memlimit: why Go services get OOM-killed in containers, and how GOMEMLIMIT
// prevents it. The program keeps `-live` MB permanently reachable and churns
// through short-lived garbage, like a busy service with a cache.
//
//	docker run --rm --memory=300m -v "$PWD":/w -w /w golang:1.26 go run ./memlimit
//	docker run --rm --memory=300m -e GOMEMLIMIT=250MiB -v "$PWD":/w -w /w golang:1.26 go run ./memlimit
package main

import (
	"flag"
	"fmt"
	"os"
	"runtime"
	"runtime/debug"
	"strings"
	"time"
)

var sink []byte // defeats escape analysis: garbage must really be allocated

func main() {
	live := flag.Int("live", 160, "MB kept alive for the whole run (the 'cache')")
	secs := flag.Int("secs", 8, "how long to run")
	flag.Parse()

	fmt.Printf("GOMEMLIMIT=%q  GOGC=%q  container limit=%s\n",
		os.Getenv("GOMEMLIMIT"), os.Getenv("GOGC"), cgroup("memory.max"))

	cache := make([][]byte, 0, *live)
	for i := 0; i < *live; i++ {
		b := make([]byte, 1<<20)
		for j := range b {
			b[j] = byte(j) // touch every page so it is really resident
		}
		cache = append(cache, b)
	}

	end := time.Now().Add(time.Duration(*secs) * time.Second)
	next := time.Now()
	for time.Now().Before(end) {
		sink = make([]byte, 4<<20) // 4 MB of garbage per iteration
		for j := 0; j < len(sink); j += 4096 {
			sink[j] = 1
		}
		if time.Now().After(next) {
			var m runtime.MemStats
			runtime.ReadMemStats(&m)
			fmt.Printf("heap=%4d MB  next GC at %4d MB  GCs=%4d  cgroup memory.current=%4d MB\n",
				m.HeapAlloc>>20, m.NextGC>>20, m.NumGC, atoiMB(cgroup("memory.current")))
			next = time.Now().Add(time.Second)
		}
	}
	runtime.KeepAlive(cache)
	fmt.Println("survived. limit from debug.SetMemoryLimit:", debug.SetMemoryLimit(-1)>>20, "MB")
}

func cgroup(file string) string {
	b, err := os.ReadFile("/sys/fs/cgroup/" + file)
	if err != nil {
		return "n/a"
	}
	return strings.TrimSpace(string(b))
}

func atoiMB(s string) int64 {
	var n int64
	fmt.Sscan(s, &n)
	return n >> 20
}

$ docker run --rm --memory=300m --memory-swap=300m -v "$PWD/bin":/b debian:bookworm /b/memlimit
GOMEMLIMIT=""  GOGC=""  container limit=314572800
heap= 164 MB  next GC at  280 MB  GCs=   6  cgroup memory.current= 173 MB
Killed
exit code: 137

$ docker run --rm --memory=300m --memory-swap=300m -e GOMEMLIMIT=250MiB -v "$PWD/bin":/b debian:bookworm /b/memlimit
heap= 164 MB  next GC at  236 MB  GCs=   6  cgroup memory.current= 173 MB
heap= 228 MB  next GC at  236 MB  GCs=1508  cgroup memory.current= 256 MB
heap= 228 MB  next GC at  236 MB  GCs=3006  cgroup memory.current= 254 MB
...
heap= 180 MB  next GC at  236 MB  GCs=10666  cgroup memory.current= 248 MB
survived. limit from debug.SetMemoryLimit: 250 MB
exit code: 0

What to notice:

Exit code 137 = 128 + 9 (SIGKILL). The kernel killed the process the moment the cgroup hit 300 MB. Go printed no error, because it never got the chance. "Next GC at 280 MB" plus runtime overhead was already too close to the limit.
With GOMEMLIMIT the target dropped to 236 MB, and memory stayed under 260 MB for the whole run.
It ran about 1,500 GCs per second. The limit traded memory for CPU. If the live heap really needs most of the limit, GOMEMLIMIT turns an OOM kill into a "death spiral" of constant GC. Go caps GC CPU at about 50% to prevent the worst of it, but the service still slows down badly. The real fix is then a bigger limit or a smaller cache.

Exercises:

Try -live 100 without GOMEMLIMIT. Does it survive? Where does memory.current peak?
Run with GODEBUG=gctrace=1 and read one line: heap before → after → live, and the GC CPU percentage.
In Kubernetes, set GOMEMLIMIT from the manifest (Chapter 82's manifest does) and alert when go_gc_duration_seconds rises sharply, which is the early warning of a death spiral.

Common mistakes

Treating every OOM kill of a Go service as a memory leak.
Setting GOMEMLIMIT equal to the container limit, which leaves no room for non-heap memory.
Setting GOGC=off with GOMEMLIMIT without understanding that the program then collects only near the limit.
Building inside the same memory-limited container that runs the service.

Check yourself

With GOGC=100 and 200 MB of live data, roughly when does the next GC run?
Why is GOMEMLIMIT called a soft limit?
What does exit code 137 tell you?