Chapter 13 — Memory allocation and the OOM killer

In one sentence

When Linux runs critically low on memory and can't free any more, it doesn't just let the system grind to a halt — it picks a process to kill, using a scoring heuristic, to save the rest of the machine.

Why this matters

"Why did my process just disappear with no error message" and "why did Kubernetes restart my pod with status OOMKilled" are the two most common ways engineers meet this mechanism for the first time, usually in production, usually stressfully.

How it actually works

   MEMORY ALLOCATION, briefly: when your program calls `malloc()` (or
   its language's equivalent -- `new`, a Python object, a Go allocation),
   it's asking a USER-SPACE allocator library for memory, which in turn
   asks the KERNEL for more address space via syscalls like `brk()` or
   `mmap()` (Chapter 2) when it needs more room. Critically: on Linux,
   by default, this request can SUCCEED even if the system doesn't
   actually have that much free memory right now -- this is called
   OVERCOMMIT. The kernel is betting that not every process will
   actually USE all the memory it asked for, all at once.

   THAT BET SOMETIMES FAILS. If enough processes actually DO try to use
   the memory they were promised, and the system genuinely runs out of
   both RAM and swap, the kernel has no way to honour the requests
   anymore. Rather than let the whole system deadlock or crash, the
   OOM KILLER (Out-Of-Memory killer) steps in and FORCIBLY KILLS A
   PROCESS to free memory.

   WHICH PROCESS GETS KILLED is decided by a "badness" score
   (`oom_score`, visible per-process in `/proc/<pid>/oom_score`), roughly
   weighted toward: how much memory a process is using (bigger =
   more likely to be picked), how long it's been running, and whether
   it's been given special treatment via `oom_score_adj` (a value from
   -1000, "never kill this," to +1000, "kill this first"). The kernel's
   goal is "free the most memory for the least collateral damage" -- but
   its notion of "damage" has no idea your database is more important
   than a stray debugging script, unless you've told it so explicitly.

Real-world example

   Kubernetes' `OOMKilled` pod status (a status any engineer running
   containers will eventually see, documented extensively in Kubernetes'
   own docs and in countless real-world troubleshooting writeups) is
   THIS EXACT MECHANISM, one layer up: a container's memory usage is
   enforced via a cgroup memory limit (Chapter 43, Phase 3's Chapter 44),
   and when a container tries to exceed it, the KERNEL'S OOM KILLER --
   scoped to that cgroup -- kills the offending process inside the
   container, and Kubernetes reports it as `OOMKilled`.
   A well-known, recurring real production pain point: the OOM killer,
   left to its DEFAULT scoring, can kill the WRONG process on a
   multi-process host or inside a multi-process container -- for
   example, killing a critical, well-behaved database process because
   it happens to have a large (but legitimate, intentional) memory
   footprint, while leaving a genuinely leaking, smaller process alive.
   This is precisely why production runbooks and platform teams commonly
   set `oom_score_adj` explicitly for critical processes (or, in
   Kubernetes, set requests/limits and Quality-of-Service classes
   deliberately, Phase 3), rather than trusting the kernel's default
   heuristic to guess correctly under pressure.

Try it yourself

# See the current OOM score for your own shell (lower = less likely to
# be killed first)
cat /proc/self/oom_score
cat /proc/self/oom_score_adj

# Watch memory overcommit in action (safe, small-scale demo)
cat /proc/sys/vm/overcommit_memory   # 0 = heuristic (default),
                                     # 1 = always allow, 2 = strict

# See what the OOM killer has done recently on your system, if anything
dmesg | grep -i "out of memory" || echo "no OOM events found (good!)"
journalctl -k | grep -i "killed process" 2>/dev/null | tail -5

   Dangerous: don't deliberately trigger the OOM killer on a shared or
   important machine. If you want to see it fire, do it on a throwaway
   VM with a small, fixed memory size, and expect to need to reboot it.

Build it in Go

Garbage-collected runtimes and memory limits interact badly by default: Go lets the heap grow to twice the live data before collecting. In Chapter 76, a Go program with 160 MB of live data is OOM-killed in a 300 MB container (exit 137) and survives with GOMEMLIMIT=250MiB.

Common mistakes

"My process crashed, it must be a bug in my code." Check dmesg/ journalctl for OOM kills FIRST — a silently-vanished process with no application-level error message is a classic OOM-kill signature.
Assuming malloc() succeeding means the memory is really available. Overcommit means a successful allocation is a PROMISE, not a guarantee — the bill can come due later, unpredictably, possibly for a DIFFERENT process entirely.
Never setting oom_score_adj or container memory limits for critical processes. Leaves you at the mercy of the kernel's generic heuristic during exactly the moment you can least afford a wrong guess.

Check yourself

What is memory overcommit, and why does the kernel do it?
What does the OOM killer do, and roughly how does it choose a victim?
Why might a critical database process get killed by the OOM killer instead of a smaller, actually-leaking process?
How does this mechanism show up in Kubernetes?