Part 13 — Performance and troubleshooting

This part pulls together CPU (Part 2), memory (Part 3), and I/O (Part 4) into the actual skill you've been building toward: reading a struggling system's vital signs correctly and knowing which of the three is really the bottleneck, instead of guessing.

Chapter 41 — Reading load average, `vmstat`, and `iostat` correctly

In one sentence

"Load average" is one of the most widely MISREAD numbers in all of Linux operations — on Linux specifically (unlike some other Unix systems), it counts processes waiting on disk I/O, not just processes waiting for CPU, and knowing that single fact prevents a huge fraction of false diagnoses.

Why this matters

Misreading load average as "pure CPU demand" sends engineers chasing CPU optimisation for problems that are actually disk or network I/O bottlenecks — a real, common, and avoidable waste of incident-response time.

How it actually works

   uptime                    # shows load average as three numbers:
                             # 1-minute, 5-minute, 15-minute averages
   # load average: 2.15, 1.98, 1.75

   WHAT THOSE NUMBERS MEAN: roughly, the average number of processes
   that were EITHER actively running on a CPU OR WAITING to (Chapter 5's
   "R" state) OR -- and this is the widely-misunderstood part, ON LINUX
   SPECIFICALLY -- in UNINTERRUPTIBLE SLEEP (Chapter 5's "D" state,
   almost always waiting on disk/network I/O), averaged over that time
   window. A load average of "4" on a 4-core machine, if it's ALL
   CPU-bound work, means the machine is fully, appropriately busy. The
   SAME number of "4," if it's mostly D-state processes stuck waiting on
   a slow, struggling disk, means something completely different --
   your CPUs might be almost entirely IDLE while load average looks
   identically "high."

   vmstat 1                    # live, per-second system stats. Key
                               # columns:
                               #   r  = processes waiting to RUN (CPU
                               #        pressure)
                               #   b  = processes BLOCKED, in
                               #        uninterruptible sleep (I/O
                               #        pressure -- Chapter 5's D state)
                               #   si/so = swap in/out (Chapter 11's
                               #        memory pressure)
                               #   us/sy/id/wa = %CPU in user time /
                               #        kernel time / idle / waiting on
                               #        I/O

   iostat -x 1                  # per-DEVICE disk I/O detail -- %util
                               # (how busy is this specific disk),
                               # await (average time per I/O request,
                               # in milliseconds -- Chapter 12's
                               # hierarchy, made concrete and
                               # measurable)

Real-world example

   Brendan Gregg -- a widely-published, well-known performance engineer
   (formerly at Netflix and Sun/Oracle, among others, and author of
   several standard reference books on this exact topic) has written
   extensively, in detailed, technical, publicly-available material, on
   exactly this chapter's core confusion: many engineers coming from
   other Unix backgrounds (or just intuition) ASSUME "load average"
   means "CPU demand," full stop -- and Linux's kernel-level definition
   has, since a very early kernel version (a specific, documented
   change from the early 1990s), DELIBERATELY included processes
   waiting on uninterruptible I/O in the count, not just CPU-runnable
   ones. Gregg's writing on the topic (searchable, still current,
   genuinely worth reading in full) has become one of the standard
   references specifically BECAUSE this misunderstanding causes REAL,
   REPEATED, DOCUMENTED misdiagnosis in production incidents: an
   engineer sees "load average: 40" on an otherwise-idle-looking
   8-core box, assumes a CPU emergency, and starts investigating the
   wrong layer entirely -- when `vmstat`'s `b` column (or a look at
   `iostat`) would have immediately shown the REAL story: a struggling
   disk or an overwhelmed network filesystem, with the CPUs sitting
   nearly idle the entire time, waiting.

Try it yourself

# Read your own machine's numbers, correctly, right now
uptime
nproc                          # how many CPU cores you actually have
                               # -- context for interpreting load average

vmstat 1 5                      # watch 'r' and 'b' columns specifically
                               # -- CPU-waiting vs I/O-waiting processes

iostat -x 1 3 2>/dev/null || sudo apt install -y sysstat  # (iostat is
                               # part of the `sysstat` package -- may
                               # need installing)

# See a CPU-bound load vs. an I/O-bound "load" side by side (safe demo)
for i in $(seq 1 4); do yes > /dev/null & done   # CPU-bound load
sleep 3; uptime; vmstat 1 2
kill $(jobs -p)                                   # clean up

Common mistakes

Treating load average as a pure CPU metric on Linux. It isn't — check vmstat's b column or iostat before assuming a "high load" number means CPU pressure specifically.
Ignoring the number of CORES when judging "is this load high." A load average of 8 is very different on a 2-core box (severely overloaded) than a 64-core one (barely used) — always read it relative to nproc.
Looking only at 1-minute load average. The three numbers together (1/5/15-minute) tell you the TREND — rising, falling, or steady — which is often more diagnostically useful than any single snapshot.

Check yourself

What does Linux's load average actually count, precisely — and how does that differ from "pure CPU demand"?
If vmstat's b column is consistently non-zero, what kind of pressure does that indicate?
Why is "load average: 8" ambiguous without also knowing the core count?

(Answers: Appendix E.)