Part 4 — I/O and storage

CPU and memory are the first two resources the OS manages on a process's behalf; I/O is the third, and the slowest by orders of magnitude. This part follows a single read() call all the way down through the kernel's I/O stack to the physical disk, and back.

Chapter 14 — The I/O stack: from `read()` to the disk platter

In one sentence

A single read() or write() call passes through many distinct layers — the C library, the kernel's virtual filesystem layer, a specific filesystem driver, the block layer, and finally a device driver — and each layer adds both useful abstraction and a small amount of real latency.

Why this matters

"I wrote to the file, why is the data still not really saved" is one of the most consequential misunderstandings in software engineering — it has caused real, well-documented data loss.

How it actually works

   THE STACK, roughly, for a file write:

     your program:      write(fd, buf, len)
                              |  (a syscall, Chapter 2)
     kernel VFS layer:  the Virtual File System -- a generic interface
                        so "write to a file" works the same whether it's
                        ext4, XFS, NFS, or a network filesystem
                              |
     filesystem driver: ext4 (or whichever) decides WHERE this data goes
                        on the underlying block device, updates its own
                        metadata (Chapter 15)
                              |
     page cache:         (Chapter 16) the write usually lands HERE first
                        -- in RAM -- and is marked "dirty," not yet
                        flushed to the physical device
                              |
     block layer:        eventually, the kernel decides to actually flush
                        dirty pages to disk; the block layer batches and
                        schedules these I/O requests (Chapter 16)
                              |
     device driver:      talks the actual protocol (SATA, NVMe, etc.) to
                        the physical device
                              |
     THE PHYSICAL DEVICE: an SSD's flash cells, or a spinning disk's
                        magnetic platter, finally, actually, stores the
                        bytes.

   THE CRITICAL, OFTEN-MISUNDERSTOOD POINT: your `write()` call RETURNING
   SUCCESSFULLY usually only means the data reached the PAGE CACHE (step
   4) -- it does NOT mean the data is safely on the physical device yet.
   If the machine loses power before the kernel flushes that dirty page,
   the write can be LOST, even though your program was told it
   succeeded. `fsync()` is the syscall that explicitly asks the kernel
   to flush a specific file's data all the way down to the device and
   wait for confirmation -- and even THEN, some storage hardware/
   virtualisation layers have historically lied about actually
   completing that flush, for performance reasons.

Real-world example

   In 2009, a real, widely-discussed controversy broke out on the Linux
   Kernel Mailing List after users of the (then-new) ext4 filesystem
   started reporting that files they had JUST WRITTEN would sometimes be
   ZERO BYTES or contain GARBAGE after an unexpected power loss or
   crash -- even though their application code had called the standard
   `write()`/`close()` sequence that had "always worked" on the older
   ext3 filesystem. The underlying cause was exactly this chapter's
   layering: ext4's newer "delayed allocation" optimisation could delay
   actually writing data to disk for LONGER than ext3 typically had,
   combined with applications that were, technically, always relying on
   an UNDOCUMENTED, filesystem-specific timing behaviour rather than
   explicitly calling `fsync()` when they actually needed a durability
   guarantee. Ext4's own lead maintainer, Ted Ts'o, and Linus Torvalds
   both weighed in publicly (their mailing-list posts from the time are
   still findable and are a genuinely excellent, blunt explanation of
   this exact layering problem) -- the eventual resolution involved BOTH
   kernel-side changes (shortening the delay) AND, more fundamentally, a
   renewed, widely-repeated industry reminder: if your application needs
   a guarantee that data is durably on disk, you must call `fsync()`
   (or use a library/database that does it correctly for you) --
   "the write() call returned" was never actually that guarantee.

Try it yourself

# See the layers in action: strace a simple file write and watch the
# actual syscalls (write, then, if the program calls it, fsync)
strace -e trace=write,fsync,close -o /tmp/trace.log \
  bash -c 'echo hello > /tmp/testfile.txt'
cat /tmp/trace.log

# Watch dirty (not-yet-flushed) pages accumulate and get flushed
cat /proc/meminfo | grep -i dirty
dd if=/dev/zero of=/tmp/bigfile bs=1M count=200 2>&1 | tail -1
cat /proc/meminfo | grep -i dirty    # likely non-zero right after
sleep 3
cat /proc/meminfo | grep -i dirty    # should shrink as the kernel flushes
rm /tmp/bigfile /tmp/testfile.txt

Common mistakes

Assuming a successful write() means the data is safe on disk. It usually only means the data reached the page cache — durability requires fsync() (or the equivalent your database/library already does for you).
Calling fsync() after every tiny write "to be safe." This defeats the performance benefit of the page cache almost entirely — the real skill is knowing WHEN durability actually matters (a committed transaction) vs. when it doesn't (an intermediate scratch file).
Blaming the filesystem for data loss that's actually an application bug. As the ext4 story shows, the real fix was usually in APPLICATIONS correctly requesting durability, not in the filesystem "misbehaving."

Check yourself

List the layers a write() call passes through, from your program to the physical device.
Does a successful write() guarantee the data is safely on disk? What does guarantee that?
What was the real, underlying lesson from the 2009 ext4 delayed-allocation controversy?