Chapter 12 — The memory hierarchy: registers to disk, and why it exists

In one sentence

Every layer of storage in a computer — CPU registers, cache, RAM, SSD, spinning disk, the network — trades capacity for speed in the opposite direction, and almost all of performance engineering is about keeping the data you need right now as high up that ladder as possible.

Why this matters

This single idea explains why caching exists at every layer of every system you'll ever build, why "just add more RAM" fixes so many problems, and why network calls are the slowest thing your code routinely does.

How it actually works

   THE LADDER, fastest/smallest at the top, slowest/largest at the
   bottom (this exact table, in this exact spirit, is widely known in
   the industry as "latency numbers every programmer should know,"
   popularised by Google engineer Jeff Dean's internal talks and now
   cited constantly across performance-engineering material):

     CPU REGISTER          ~ sub-nanosecond          (a handful of bytes)
     L1 CPU CACHE          ~ 1 nanosecond             (tens of KB)
     L2 CPU CACHE          ~ few nanoseconds          (hundreds of KB)
     L3 CPU CACHE          ~ tens of nanoseconds      (tens of MB)
     MAIN MEMORY (RAM)     ~ 100 nanoseconds          (GBs)
     SSD (random read)     ~ 100 MICROSECONDS         (hundreds of GB-TB)
     SPINNING DISK (seek)  ~ 1-10 MILLISECONDS        (many TB)
     NETWORK (same DC)     ~ 0.5 MILLISECONDS round trip
     NETWORK (cross-region/
       continent)          ~ 50-150+ MILLISECONDS round trip

   NOTICE THE GAPS: RAM is roughly 100x slower than L1 cache. An SSD is
   roughly 1,000x slower than RAM. A cross-region network call can be
   MILLIONS of times slower than a register access. These aren't small
   differences -- they're differences of many ORDERS OF MAGNITUDE, and
   they compound: a program that "just" does one unnecessary disk read
   per request, at scale, can dominate your entire system's latency
   budget.

   THIS IS WHY CACHING EXISTS EVERYWHERE: the CPU caches recently-used
   RAM contents (hardware-managed, invisible to you). The OS caches
   recently-read disk blocks in spare RAM (Chapter 16's page cache).
   Your application caches expensive database queries in Redis. Your
   CDN caches HTTP responses close to users. It's the SAME PATTERN,
   repeated at every layer of the stack, because the SAME physics
   (this ladder) applies at every layer.

Real-world example

   This exact table -- Jeff Dean's "Numbers Every Programmer Should
   Know" -- has circulated across the software industry for over a
   decade (originally from internal Google engineering talks, widely
   re-published and interactively visualised, e.g. at
   `colin-scott.github.io/personal_website/research/interactive_latency.html`)
   specifically because engineers KEPT making the same class of mistake:
   architecting a system as if a network call, a disk read, and a memory
   access all cost roughly the same. A single unnecessary synchronous
   network call inside a hot loop -- something that looks completely
   harmless in code review -- can be the difference between a service
   responding in single-digit milliseconds and one that takes whole
   seconds, purely because of where "network round trip" sits on this
   ladder relative to everything above it. Real, well-known performance
   incidents at scale are very often traced back to exactly this: doing
   an operation from a slow rung of the ladder when a faster one would
   have worked, repeated enough times (in a loop, per-request, per-item)
   for the multiplier to dominate everything else.

Try it yourself

# Feel the RAM-vs-disk gap directly: write and read the same amount of
# data via RAM (tmpfs, usually mounted at /dev/shm) vs actual disk
dd if=/dev/zero of=/dev/shm/testfile bs=1M count=512 2>&1 | tail -1
dd if=/dev/zero of=./testfile-ondisk bs=1M count=512 2>&1 | tail -1
rm /dev/shm/testfile ./testfile-ondisk

# See your CPU's actual cache sizes
lscpu | grep -i cache

Common mistakes

"RAM is basically infinite/free compared to disk, so it doesn't matter." The GAPS at every level of this ladder matter, not just the RAM/disk one — cache-unfriendly code (poor memory access patterns) can be dramatically slower even when everything technically "fits in RAM."
Treating all "fast" operations as equally fast. A local network call and a cross-region network call can differ by 100x or more — "it's just a network call" hides an enormous range.
Over-caching without eviction strategy. A cache with no size limit or eviction policy just becomes a slow memory leak wearing a fast hat.

Check yourself

List the memory hierarchy from fastest/smallest to slowest/largest.
Roughly how many orders of magnitude slower is a disk read than a RAM access?
Why does the SAME caching pattern appear at the CPU, OS, application, and CDN layers?