Part 18 -- Capstone projects

Reading about OS internals, Linux commands, and Kubernetes gets you to "I understand this." Building something end to end, and then deliberately breaking it and fixing it, gets you to "I can do this under pressure," which is the actual bar for a beginner SRE or backend engineer role. These four projects are designed to be done in order -- each one assumes the skills from the previous one and the previous parts of this guide.

There is no single "correct" solution to any of these. What matters is that you can explain every decision you made, in the vocabulary this guide has given you: syscalls, page cache, cgroups limits, readiness probes, image layers.

Chapter 62 -- Project 1: build and profile a small program, OS-level

In one sentence

Write a small program that does real work, then use the OS-level tools from Parts 1-6 (strace, time, /proc, ulimit, top) to explain, in kernel terms, exactly where its time and memory go.

Why this matters

This project exists to make the abstract concrete: "page fault," "context switch," and "syscall overhead" stop being vocabulary words and become numbers you personally measured on your own machine. This is also close to the actual daily work of performance engineering -- "why is this slow" is answered by measurement, not by guessing, and the tools here are the same ones used to debug production latency issues at any company running Linux.

How it actually works

The project, concretely:

   STEP 1 -- Write a program (any language) that:
     a) reads a large text file (a few hundred MB -- e.g. concatenate
        some log files, or download a public dataset)
     b) counts word frequency (a classic, CPU + I/O mixed workload)
     c) writes the top 20 words to stdout

   STEP 2 -- Measure it plainly first:
     /usr/bin/time -v ./wordcount bigfile.txt
     Read every line of the output: elapsed time, user time, system
     time, maximum resident set size, major/minor page faults,
     voluntary/involuntary context switches. Write down what each
     number means in your own words, using Chapters 4-13's vocabulary.

   STEP 3 -- Trace its syscalls:
     strace -c ./wordcount bigfile.txt
     Which syscall dominates the count? Which dominates total time?
     (Usually `read` calls dominate count; for a naive line-by-line
     reader, this can be enormous -- tens of thousands of small reads
     instead of a handful of large ones.)

   STEP 4 -- Change the I/O strategy and re-measure:
     Rewrite the file-reading part to read in large chunks (e.g. 1MB
     buffers) instead of line-by-line with an unbuffered reader, or
     use memory-mapping (mmap) instead of read(). Re-run STEP 2 and
     STEP 3. Compare: syscall count, elapsed time, major page faults.

   STEP 5 -- Constrain it with cgroups and watch it suffer honestly:
     systemd-run --scope -p MemoryMax=50M ./wordcount hugefile.txt
     If the file is bigger than 50MB and your program loads it all
     into memory at once, watch it get OOM-killed. Check `dmesg` or
     `journalctl -k` for the OOM killer's log line (Chapter 11).
     Then fix the program to stream the file instead of loading it
     all at once, and show it now completes under the same limit.

Real-world example

This exact exercise -- profile, find the surprising bottleneck, fix it, re-measure -- mirrors one of the most famous real performance investigations in systems literature: the discovery that naive line-by-line I/O (calling read() or getline() once per line) can be 10-100x slower than buffered, chunked reads, purely because of syscall overhead, not because of disk speed. This is precisely the kind of finding Brendan Gregg documents extensively in his USE Method and "Systems Performance" methodology (used at Netflix, and taught widely in the SRE community): most performance problems are not "the CPU is too slow," they are "the program is making the kernel do far more work than the task actually requires," and the only way to see that is to measure syscalls and page faults directly instead of guessing from wall-clock time alone.

Try it yourself

This chapter's project is the "try it yourself" -- work through Steps 1 through 5 above on your own machine (a Linux VM or container is fine; systemd-run cgroup limiting won't work in plain macOS/Windows, but Docker's --memory flag gives you the equivalent, see Chapter 47).

Common mistakes

Measuring only wall-clock time (time ./program) and stopping there -- wall-clock time alone doesn't tell you why it's slow, only that it is.
Optimizing before measuring -- rewriting code based on a guess about what's slow, then finding the real bottleneck was somewhere else entirely.
Running the "before" and "after" measurements under different conditions (different file, different machine load, cache warm vs cold) and drawing conclusions from a comparison that isn't apples-to-apples. Run each version multiple times and note whether the file is already in the page cache (Chapter 15) from a previous run.

Check yourself

If /usr/bin/time -v shows a large number of "minor" page faults but zero "major" page faults, what does that tell you, and should you be worried?
Why would switching from line-by-line reads to 1MB chunked reads reduce the syscall count by orders of magnitude, and why does that matter even though both approaches read the same total bytes?
Your program gets OOM-killed under a 50MB memory limit even though the input file is only 30MB. What are two plausible reasons memory usage could exceed the file size?