How it actually works

   WITHOUT AN OS: your program would need to know EXACTLY which physical
   RAM address is free, EXACTLY how to talk to this specific disk
   controller, and EXACTLY how to avoid another program overwriting its
   memory at the same instant. Every program would need to be its own
   tiny operating system. This was, roughly, how computing worked before
   the 1960s-70s: one program, one computer, no sharing.

   WITH AN OS, three lies are told to every program, consistently:
     1. "You have the CPU to yourself."
        (Reality: the scheduler, Chapter 8, is switching between you and
         every other runnable program many times per second.)
     2. "You have a huge, private block of memory, starting at address 0."
        (Reality: virtual memory, Chapter 10, maps your program's private
         address space onto scattered, shared physical RAM -- and part of
         it might not even be in RAM right now.)
     3. "Files and devices are just streams of bytes you can read/write."
        (Reality: a "file" might be on a spinning disk, an SSD, a network
         share, or not even really exist, like /dev/random.)

   The OS maintains these lies by being the ONLY thing with direct
   hardware access (Chapter 2) and mediating every request. This is what
   "operating system" means: it OPERATES the hardware ON BEHALF of every
   program that wants to use it.

How it actually works

   THE CPU ITSELF enforces this boundary, via "privilege rings" (or
   "modes") built into the hardware -- it's not just a software policy
   the kernel chooses to follow, it's physically enforced by the
   processor:
     RING 0 / KERNEL MODE   full access: can execute any instruction,
                            touch any memory, talk to any device.
     RING 3 / USER MODE     restricted: cannot execute privileged
                            instructions or touch hardware directly.
   Your shell, your text editor, your web server -- all of it runs in
   user mode. Only the kernel runs in kernel mode.

   SO HOW DOES A PROGRAM READ A FILE, IF IT CAN'T TOUCH THE DISK?
   It makes a SYSTEM CALL (a "syscall"): a formal request to the kernel,
   using a special CPU instruction that deliberately, safely switches the
   processor from user mode to kernel mode, runs a specific bit of kernel
   code to service the request, and switches back.

     your program:  open("data.txt", O_RDONLY)
                         |  (this C library function wraps a syscall)
                         v
     CPU switches to kernel mode (a "trap")
                         |
     kernel: checks permissions, finds the file, sets up a file
             descriptor (Chapter 23), switches back to user mode
                         |
     your program: receives a file descriptor number, continues

   THIS IS SLOWER than a plain function call -- crossing the boundary has
   real, measurable overhead (a "context switch," Chapter 9) -- which is
   exactly why well-written programs try to minimise the NUMBER of
   syscalls they make (buffering writes instead of writing one byte at a
   time, for example).

The syscalls you'll meet constantly, so the names stop being scary:

   read() / write()      move bytes between a program and a file/socket
   open() / close()      get/release a file descriptor
   fork() / exec()       create a new process / replace a process's code
                        (Chapter 6)
   mmap()                 map a file or memory region into your address
                        space directly
   socket() / connect()   networking (a socket is just another kind of
                        file descriptor)
   wait()                 a parent process waiting for a child to finish

How it actually works

   1. YOU TYPE: ./myapp arg1 arg2

   2. YOUR SHELL calls fork() (Chapter 6) -- the kernel clones the
      shell's own process into a near-identical CHILD process.

   3. THE CHILD calls exec() -- this REPLACES the child's code and
      memory with `myapp`'s code, but keeps the same process ID. The
      kernel loads the executable from disk, sets up its memory layout
      (Chapter 10), and starts running its `main()` function.

   4. THE PROCESS RUNS, doing whatever it does, making syscalls
      (Chapter 2) whenever it needs the kernel's help, getting scheduled
      on and off the CPU (Chapter 8) the whole time.

   5. THE PROCESS ENDS, one of three ways:
        - it returns from main() / calls exit(N) deliberately
        - it's killed by a SIGNAL (Chapter 20) -- e.g. you pressed
          Ctrl-C, or something sent SIGKILL
        - it crashes (an illegal memory access, division by zero, etc.
          -- the kernel delivers this as a signal too)
      Either way, the kernel records an EXIT STATUS: a small number, by
      convention 0 for "success" and non-zero for "something went wrong"
      (the SPECIFIC non-zero value is up to the program).

   6. THE PARENT (your shell) can call wait() to retrieve that exit
      status. Until it does, the finished process sits in the process
      table as a ZOMBIE -- it's done running, but the kernel keeps its
      exit status around until the parent asks for it. (A parent that
      never calls wait() leaks zombies -- Chapter 5 covers this properly.)

   $ ./myapp arg1 arg2
   $ echo $?              # the shell's own wait() already happened;
                          # $? holds the exit status it collected
   0

How it actually works

   A process is NOT just "the code running." The kernel maintains a
   whole record for it (a "process control block") including:
     PID           a unique process ID number
     MEMORY MAP     its private virtual address space (Chapter 10)
     FILE
     DESCRIPTORS    a table of everything it has open (files, sockets,
                    pipes -- Chapter 23)
     REGISTERS      the CPU's exact state when this process was last
                    paused, so it can resume perfectly
     PARENT PID     who created it (for wait()/zombies, Chapter 3)
     PERMISSIONS     which user it runs as, what it's allowed to do
     SIGNAL HANDLERS custom behaviour for signals it's registered for
                    (Chapter 20)

   Creating a process is NOT CHEAP: the kernel has to allocate and set up
   all of the above. This cost is exactly why the industry has spent
   decades inventing lighter-weight alternatives -- threads (Chapter 7),
   thread pools, event loops, and eventually containers (Phase 3), all of
   which exist partly to avoid paying the full cost of a brand-new
   process for every unit of work.

How it actually works

   THE CORE STATES (Linux ps/top codes in parentheses):
     RUNNING (R)          actually executing on a CPU right now, OR
                          ready and waiting for the scheduler to give it
                          a turn (both are shown as "R" -- "runnable").
     SLEEPING /
     INTERRUPTIBLE (S)     waiting for something (input, a timer, a lock)
                          and CAN be woken early by a signal. This is
                          the state of almost every idle, well-behaved
                          process on your machine right now.
     UNINTERRUPTIBLE
     SLEEP (D)             waiting for something -- usually disk or
                          network I/O -- and CANNOT be interrupted, not
                          even by SIGKILL, until the I/O completes or
                          times out. A process stuck here for a long
                          time is one of the most frustrating real
                          production problems (see below).
     STOPPED (T)           paused deliberately (Ctrl-Z, or SIGSTOP) --
                          not running, not scheduled, just frozen until
                          resumed (SIGCONT).
     ZOMBIE (Z)            finished running; just waiting for its parent
                          to collect its exit status (Chapter 3).

   THE TRANSITIONS, roughly:
     new process -> RUNNING (well, "runnable" -- waiting its turn)
     RUNNING -> SLEEPING     (it asked for I/O, or called sleep())
     SLEEPING -> RUNNING     (the thing it was waiting for happened)
     RUNNING -> STOPPED      (a stop signal arrived)
     STOPPED -> RUNNING      (a continue signal arrived)
     RUNNING -> ZOMBIE       (it exited)
     ZOMBIE -> gone          (the parent called wait())

How it actually works

   fork()   makes an almost-exact COPY of the calling process: same code,
            same memory contents (initially), same open file descriptors
            -- but a NEW PID. Both the original (parent) and the copy
            (child) continue running from the EXACT SAME LINE of code,
            right after the fork() call, and can tell themselves apart
            by fork()'s RETURN VALUE:
              in the parent:  fork() returns the child's PID
              in the child:   fork() returns 0
              (if it fails:   fork() returns -1)

   exec()   REPLACES the calling process's code and memory with a
            DIFFERENT program entirely, keeping the same PID and open
            file descriptors. It doesn't return on success -- the calling
            program is simply gone, replaced.

   WHY TWO SEPARATE STEPS, rather than one "run this other program" call?
   Because fork() ALONE is useful too -- and because the gap BETWEEN
   fork() and exec() is exactly where a shell does useful setup work
   (redirecting a file descriptor for `>`, changing directory, setting
   environment variables) on the CHILD ONLY, before replacing it, without
   ever affecting the parent shell itself:

     pid_t pid = fork();
     if (pid == 0) {
         // child: I am a copy of the shell right now
         // ... redirect stdout to a file, if the user wrote `> out.txt`
         // ... then:
         execve("/bin/ls", args, environ);   // now I become `ls`
     } else {
         // parent (the shell): keep running, maybe wait() for the child
     }

How it actually works

   A PROCESS can contain ONE OR MORE threads. All threads in the same
   process share:
     - the same memory (global variables, heap allocations)
     - the same open file descriptors
     - the same process ID (from the OS's perspective, mostly)

   Each thread gets its OWN:
     - stack (local variables, function call history)
     - CPU register state / program counter (where THIS thread currently
       is in the code)
     - thread ID

   WHY THIS MATTERS SO MUCH: because threads share memory, TWO THREADS
   CAN TOUCH THE SAME VARIABLE AT THE SAME TIME -- something two
   separate PROCESSES simply cannot do by accident, because their
   memory is isolated (Chapter 10). This is BOTH the main reason threads
   are useful (fast communication, no copying needed, no fork() cost)
   AND the main reason threaded code is notoriously hard to get right
   (Chapter 17's race conditions, Chapter 18's locks).

   THREADS VS. PROCESSES, the trade-off in one table:
     PROCESSES   isolated (safer), heavier to create, communicate via
                slower IPC mechanisms (Chapter 19)
     THREADS     share memory (faster, riskier), cheap to create, need
                explicit synchronisation (Chapter 18) to be safe

How it actually works

   THE BASIC PROBLEM: you might have 4 CPU cores and 200 runnable
   processes. The scheduler has to pick which handful run RIGHT NOW, and
   for how long, before switching to the next batch -- fairly enough that
   nothing starves, responsively enough that interactive programs (your
   mouse cursor, your SSH session) don't feel laggy, and efficiently
   enough that the switching itself (Chapter 9) doesn't eat all your
   CPU.

   LINUX'S SCHEDULER (CFS -- the Completely Fair Scheduler, the default
   since 2007, itself succeeded by EEVDF in newer kernels) works, in
   spirit, like this:
     - every runnable task accumulates "virtual runtime" while it runs
     - the scheduler always picks the task with the LEAST virtual
       runtime to run next -- i.e., whoever has had the LEAST CPU time
       so far, relatively speaking, goes next.
     - this naturally converges toward fairness: no task can hog the CPU
       indefinitely, because the more it runs, the further behind
       (in scheduling priority) it falls relative to everyone else.

   NICE VALUES let you bias this, without breaking fairness entirely:
     `nice -n 10 mycommand`    lower priority (be nice to others,
                              range -20 to +19, higher number = lower
                              priority)
     `nice -n -10 mycommand`   higher priority (needs privilege)
   This doesn't GUARANTEE a task runs sooner -- it changes the WEIGHT the
   scheduler uses when deciding, so a "niced-up" process accumulates
   virtual runtime slower, and gets picked more often, relatively.

How it actually works

   AN INTERRUPT is a signal from HARDWARE (a network card just received
   a packet, a disk finished a read, a timer fired) telling the CPU
   "stop what you're doing and handle this NOW." The CPU pauses whatever
   it was running, jumps to a small piece of kernel code (an interrupt
   handler) to deal with it, then resumes.

   A CONTEXT SWITCH is the broader mechanism: the kernel saves the FULL
   state of whatever was running (every CPU register, the program
   counter, memory-mapping information) into that process/thread's
   saved state, and loads a DIFFERENT one's saved state instead so IT
   can run. This happens:
     - when the scheduler (Chapter 8) decides it's someone else's turn
     - when a process makes a blocking syscall (Chapter 2) and has to
       wait
     - when a hardware interrupt needs kernel attention

   WHY IT'S NOT FREE:
     - saving/restoring register state takes real CPU cycles
     - it very likely INVALIDATES THE CPU CACHE (0.4's memory ladder) --
       the new process's data isn't in the fast, nearby cache, so its
       first few memory accesses are slow, cache-miss reads from RAM
     - on modern CPUs, it can also flush speculative-execution and
       branch-prediction state, adding further real overhead

   A rough, often-cited order of magnitude: a context switch costs low
   MICROSECONDS -- a few thousand CPU cycles' worth of pure overhead,
   before the newly-scheduled task does a single second of useful work.
   That sounds tiny, until you're doing it hundreds of thousands of
   times per second under heavy load.

How it actually works

   WITHOUT VIRTUAL MEMORY, every program would need to know the actual,
   physical RAM address of everything it touches -- and if two programs
   both tried to use the same physical address, disaster. This was a
   real problem in early computing.

   WITH VIRTUAL MEMORY, every process gets its OWN, PRIVATE address
   space -- typically the full range a 64-bit pointer can express. When
   your program reads or writes address `0x7fff1234`, the CPU's MEMORY
   MANAGEMENT UNIT (MMU) -- dedicated hardware, working with page tables
   the kernel maintains -- translates that VIRTUAL address into whatever
   PHYSICAL RAM address it actually corresponds to, if any, completely
   transparently to your program:

     Process A's virtual 0x1000  ->  physical RAM address 0x7A2000
     Process B's virtual 0x1000  ->  physical RAM address 0x3F1000
                                     (a DIFFERENT physical location --
                                      this is the whole isolation
                                      guarantee)

   THIS ALSO ENABLES SOME GENUINELY USEFUL TRICKS:
     - two processes can SHARE the same physical memory (a shared
       library like libc) while each believes it has its own private
       copy at its own virtual address.
     - a virtual address doesn't need to be backed by RAM AT ALL, right
       now -- it can be backed by disk (Chapter 11's paging), or not
       backed by anything until it's actually touched (this is how
       `malloc()`-ing a huge amount of memory can succeed instantly:
       nothing physical is allocated until you actually write to it).

How it actually works

   PAGES: physical RAM and virtual address space are both divided into
   fixed-size chunks -- PAGES -- commonly 4KB on Linux/x86 (larger "huge
   pages," 2MB or 1GB, exist for specific performance-sensitive
   workloads). The kernel's PAGE TABLE records, for every virtual page a
   process uses, WHERE (or whether) it's currently backed by a physical
   page.

   A PAGE FAULT happens when a process accesses a virtual address whose
   page ISN'T currently mapped to physical RAM. There are several very
   different flavours:

     MINOR FAULT (cheap, common, totally normal)
       the page exists somewhere accessible but isn't yet mapped into
       THIS process (e.g. a shared library page another process already
       loaded) -- the kernel just updates the mapping. Fast, invisible.

     MAJOR FAULT (expensive)
       the data has to be read FROM DISK -- either it's part of a
       memory-mapped file not yet loaded, or (the painful case) it was
       previously SWAPPED OUT to disk to free up RAM for something else,
       and now has to be swapped back IN. This is orders of magnitude
       slower than RAM access (Chapter 12's hierarchy).

     SEGMENTATION FAULT (fatal)
       the process touched a virtual address that ISN'T VALID AT ALL for
       it -- not "not yet loaded," but genuinely not part of its address
       space (a wild pointer, a buffer overrun). The kernel delivers
       SIGSEGV (Chapter 20) and the process usually dies. This is the
       infamous "segfault."

   THE TLB (Translation Lookaside Buffer) is a small, very fast CACHE,
   right on the CPU, of recent virtual-to-physical translations -- so the
   full page-table lookup (itself potentially several memory accesses)
   doesn't have to happen on EVERY single memory access. A "TLB miss"
   means falling back to the slower, full page-table walk.

How it actually works

   THE LADDER, fastest/smallest at the top, slowest/largest at the
   bottom (this exact table, in this exact spirit, is widely known in
   the industry as "latency numbers every programmer should know,"
   popularised by Google engineer Jeff Dean's internal talks and now
   cited constantly across performance-engineering material):

     CPU REGISTER          ~ sub-nanosecond          (a handful of bytes)
     L1 CPU CACHE          ~ 1 nanosecond             (tens of KB)
     L2 CPU CACHE          ~ few nanoseconds          (hundreds of KB)
     L3 CPU CACHE          ~ tens of nanoseconds      (tens of MB)
     MAIN MEMORY (RAM)     ~ 100 nanoseconds          (GBs)
     SSD (random read)     ~ 100 MICROSECONDS         (hundreds of GB-TB)
     SPINNING DISK (seek)  ~ 1-10 MILLISECONDS        (many TB)
     NETWORK (same DC)     ~ 0.5 MILLISECONDS round trip
     NETWORK (cross-region/
       continent)          ~ 50-150+ MILLISECONDS round trip

   NOTICE THE GAPS: RAM is roughly 100x slower than L1 cache. An SSD is
   roughly 1,000x slower than RAM. A cross-region network call can be
   MILLIONS of times slower than a register access. These aren't small
   differences -- they're differences of many ORDERS OF MAGNITUDE, and
   they compound: a program that "just" does one unnecessary disk read
   per request, at scale, can dominate your entire system's latency
   budget.

   THIS IS WHY CACHING EXISTS EVERYWHERE: the CPU caches recently-used
   RAM contents (hardware-managed, invisible to you). The OS caches
   recently-read disk blocks in spare RAM (Chapter 16's page cache).
   Your application caches expensive database queries in Redis. Your
   CDN caches HTTP responses close to users. It's the SAME PATTERN,
   repeated at every layer of the stack, because the SAME physics
   (this ladder) applies at every layer.

How it actually works

   MEMORY ALLOCATION, briefly: when your program calls `malloc()` (or
   its language's equivalent -- `new`, a Python object, a Go allocation),
   it's asking a USER-SPACE allocator library for memory, which in turn
   asks the KERNEL for more address space via syscalls like `brk()` or
   `mmap()` (Chapter 2) when it needs more room. Critically: on Linux,
   by default, this request can SUCCEED even if the system doesn't
   actually have that much free memory right now -- this is called
   OVERCOMMIT. The kernel is betting that not every process will
   actually USE all the memory it asked for, all at once.

   THAT BET SOMETIMES FAILS. If enough processes actually DO try to use
   the memory they were promised, and the system genuinely runs out of
   both RAM and swap, the kernel has no way to honour the requests
   anymore. Rather than let the whole system deadlock or crash, the
   OOM KILLER (Out-Of-Memory killer) steps in and FORCIBLY KILLS A
   PROCESS to free memory.

   WHICH PROCESS GETS KILLED is decided by a "badness" score
   (`oom_score`, visible per-process in `/proc/<pid>/oom_score`), roughly
   weighted toward: how much memory a process is using (bigger =
   more likely to be picked), how long it's been running, and whether
   it's been given special treatment via `oom_score_adj` (a value from
   -1000, "never kill this," to +1000, "kill this first"). The kernel's
   goal is "free the most memory for the least collateral damage" -- but
   its notion of "damage" has no idea your database is more important
   than a stray debugging script, unless you've told it so explicitly.

How it actually works

   THE STACK, roughly, for a file write:

     your program:      write(fd, buf, len)
                              |  (a syscall, Chapter 2)
     kernel VFS layer:  the Virtual File System -- a generic interface
                        so "write to a file" works the same whether it's
                        ext4, XFS, NFS, or a network filesystem
                              |
     filesystem driver: ext4 (or whichever) decides WHERE this data goes
                        on the underlying block device, updates its own
                        metadata (Chapter 15)
                              |
     page cache:         (Chapter 16) the write usually lands HERE first
                        -- in RAM -- and is marked "dirty," not yet
                        flushed to the physical device
                              |
     block layer:        eventually, the kernel decides to actually flush
                        dirty pages to disk; the block layer batches and
                        schedules these I/O requests (Chapter 16)
                              |
     device driver:      talks the actual protocol (SATA, NVMe, etc.) to
                        the physical device
                              |
     THE PHYSICAL DEVICE: an SSD's flash cells, or a spinning disk's
                        magnetic platter, finally, actually, stores the
                        bytes.

   THE CRITICAL, OFTEN-MISUNDERSTOOD POINT: your `write()` call RETURNING
   SUCCESSFULLY usually only means the data reached the PAGE CACHE (step
   4) -- it does NOT mean the data is safely on the physical device yet.
   If the machine loses power before the kernel flushes that dirty page,
   the write can be LOST, even though your program was told it
   succeeded. `fsync()` is the syscall that explicitly asks the kernel
   to flush a specific file's data all the way down to the device and
   wait for confirmation -- and even THEN, some storage hardware/
   virtualisation layers have historically lied about actually
   completing that flush, for performance reasons.

How it actually works

   THREE SEPARATE THINGS, commonly conflated as "a file":

     THE DATA           the actual bytes, stored in disk blocks
                        somewhere.
     THE INODE          a fixed-size metadata record: size, permissions,
                        owner, timestamps, and (crucially) POINTERS to
                        which disk blocks hold the data. Identified by an
                        INODE NUMBER (see it with `ls -i`).
     THE DIRECTORY
     ENTRY               just a NAME mapped to an inode number, living
                        inside a directory (which is itself just a
                        special kind of file). Multiple directory entries
                        (even in different directories) CAN point to the
                        SAME inode -- this is exactly what a HARD LINK
                        is (Chapter 25).

   THIS EXPLAINS SEVERAL THINGS THAT OTHERWISE SEEM WEIRD:
     - deleting a file (`rm`) doesn't necessarily delete the DATA -- it
       removes a directory entry and DECREMENTS the inode's "link
       count." The data is only actually freed when the link count AND
       the number of processes with the file open both reach zero. This
       is why a huge log file, deleted while a process still has it
       open, doesn't free disk space until that process closes it or
       restarts.
     - renaming/moving a file WITHIN the same filesystem is nearly
       instant, regardless of file size -- it's just updating a
       directory entry to point at the same inode, no data is actually
       moved. (Moving ACROSS filesystems is different -- it has to
       actually copy the data, then delete the original.)
     - a filesystem can run out of INODES before it runs out of DATA
       SPACE -- each filesystem has a FIXED number of inodes, set when
       it was created, independent of how much raw storage capacity
       exists.

   JOURNALING (ext4, XFS, and most modern filesystems use it): before
   making a metadata change, the filesystem first writes a small record
   of its INTENT to a journal (a dedicated log area). If the system
   crashes mid-operation, the filesystem can REPLAY the journal on
   next boot to finish or cleanly roll back the interrupted operation,
   instead of leaving the filesystem's own internal structures in an
   inconsistent, possibly-corrupted state. This is why modern Linux
   boxes almost never need the old, slow, full filesystem-consistency
   check after an unclean shutdown that was routine decades ago.

How it actually works

   THE PAGE CACHE: whenever the kernel reads data from disk, it keeps a
   copy in RAM (in pages it isn't otherwise using) in case that same
   data is needed again soon -- which, for most real workloads, it often
   is (re-reading the same config file, the same popular database rows,
   the same commonly-loaded library). This is transparent to your
   program: you didn't ask for it, you don't manage it, but it makes
   repeated reads dramatically faster (Chapter 12's hierarchy: RAM vs.
   disk).

   BUFFERING WRITES works similarly, in the other direction (Chapter
   14): writes typically land in the page cache first (as "dirty"
   pages) and get flushed to the actual device by the kernel on its own
   schedule (or when explicitly `fsync()`-ed), batching and reordering
   them for efficiency rather than hitting the disk on every single
   write.

   THE KEY INSIGHT FOR READING `free -h` CORRECTLY: this cached data is
   NOT "wasted" or "unavailable" memory. The kernel will INSTANTLY
   reclaim page-cache pages the moment any process actually needs that
   RAM for something else -- there's no cost or delay to giving it back
   (unless it's DIRTY and needs flushing first). This is why modern
   `free -h` output distinguishes:
     "free"       truly, completely unused memory (often deliberately
                 near zero on a healthy, long-running Linux box -- the
                 kernel would rather use spare RAM for cache than leave
                 it idle)
     "available"  the number that actually matters: free memory PLUS
                 memory that could be reclaimed painlessly from cache if
                 needed -- this is "how much RAM can a new process
                 actually use right now."

   DISK SCHEDULING: the kernel's block layer (Chapter 14) doesn't just
   send I/O requests to the device in the order they arrive -- it can
   reorder and batch them (an I/O SCHEDULER) to reduce costly disk-head
   movement (on spinning disks) or maximise the parallel request queue
   depth (on SSDs/NVMe, which have very different performance
   characteristics and mostly use simpler, low-overhead schedulers like
   `mq-deadline` or `none` rather than the seek-minimising algorithms
   designed for spinning disks).

How it actually works

   THE OS MAKES FEW TIMING GUARANTEES. When two threads (Chapter 7) or
   processes run "at the same time," the scheduler (Chapter 8) decides
   the EXACT interleaving of their instructions, and that interleaving
   can change from run to run -- based on system load, CPU count,
   scheduling decisions that have nothing to do with your program's
   logic.

   THE CLASSIC SHAPE OF A RACE CONDITION -- a "check then act" sequence
   that ISN'T ATOMIC (isn't guaranteed to happen as one uninterruptible
   step):

     Thread A: reads shared_counter (sees 5)
     Thread B: reads shared_counter (ALSO sees 5, before A writes back)
     Thread A: writes shared_counter = 5 + 1 = 6
     Thread B: writes shared_counter = 5 + 1 = 6      <- should be 7!

   Both threads read the SAME starting value because the read-modify-
   write wasn't protected -- one of the two increments is SILENTLY LOST.
   This can happen with shared memory (Chapter 7's threads), with
   files (two processes both checking "does this file exist" then
   both creating it), or with any shared, mutable resource touched from
   more than one thread of control without coordination.

   THIS IS WHY THIS GUIDE KEEPS RETURNING TO "ATOMIC" OPERATIONS
   (Chapter 18's locks, and this same theme appears throughout the
   companion security guide's race-condition chapter, in a web-request
   context) -- the fix is always some mechanism that makes the
   check-and-act sequence happen as ONE step nothing else can interleave
   with.

How it actually works

   A MUTEX (mutual exclusion lock) is the simplest form: only ONE
   thread/process can HOLD it at a time. Anyone else trying to acquire
   it BLOCKS (waits) until the holder releases it.

     lock(mutex)
       counter += 1     // now genuinely safe: nobody else can be in
                        // here at the same time
     unlock(mutex)

   A SEMAPHORE generalises this: it allows up to N holders at once
   (a mutex is just a semaphore with N=1) -- useful for things like
   "allow at most 5 concurrent connections to this resource."

   DEADLOCK is the classic failure mode of locking: two (or more) threads
   each hold a lock the OTHER one needs, and neither will ever release
   what it has:

     Thread A:  locks Resource1, then tries to lock Resource2 (blocked --
                B has it)
     Thread B:  locks Resource2, then tries to lock Resource1 (blocked --
                A has it)
     -> both wait FOREVER. Nothing will ever unblock either of them.

   FOUR CONDITIONS must ALL be true for deadlock to occur (the classic
   formulation, from Coffman et al., 1971) -- and breaking ANY ONE of
   them prevents it:
     1. MUTUAL EXCLUSION       resources can't be shared (inherent to
                              locks)
     2. HOLD AND WAIT          a thread holds one resource while waiting
                              for another
     3. NO PREEMPTION          a resource can't be forcibly taken away
     4. CIRCULAR WAIT          a cycle of threads each waiting on the
                              next

   THE MOST COMMON REAL-WORLD FIX: always acquire multiple locks in the
   SAME, CONSISTENT ORDER, everywhere in your codebase (e.g. always lock
   "lower resource ID before higher resource ID") -- this breaks
   condition 4, circular wait, and is simple enough to actually follow
   as a team-wide convention.

How it actually works

   PIPES: a one-directional, in-kernel byte stream connecting two
   processes' file descriptors (Chapter 23) -- one writes, the other
   reads, and the kernel buffers the data between them. This is EXACTLY
   what `|` does in your shell:

     ls -la | grep ".txt"
     the shell: creates a pipe, forks two children (Chapter 6), connects
     `ls`'s stdout to the pipe's write end and `grep`'s stdin to the
     pipe's read end. `ls` never needs to know `grep` exists, or vice
     versa -- they just read/write file descriptors, and the KERNEL
     moves the bytes between them.

   Named pipes (FIFOs) work the same way but have a filesystem path,
   so unrelated processes (not just parent/child) can connect to them.

   SOCKETS: like pipes, but bidirectional and, crucially, can connect
   processes on DIFFERENT MACHINES (network sockets, TCP/UDP) or on
   the SAME machine (Unix domain sockets -- faster than network sockets
   for local-only communication, since they skip the whole network
   stack).

   SHARED MEMORY: the fastest IPC mechanism, because it skips COPYING
   data through the kernel entirely -- two processes map the SAME
   physical memory pages (Chapter 10) into their own virtual address
   spaces, and can read/write it directly. Fastest, but the RISKIEST:
   now you need explicit synchronisation (Chapter 18's locks) between
   processes, not just threads, because the OS gives you NO protection
   against both writing at once.

   MESSAGE QUEUES: the kernel maintains a queue of discrete MESSAGES
   (not a raw byte stream like a pipe) that processes can send to and
   receive from, with some ordering/priority semantics -- less commonly
   used directly today than the other three, but still real and
   available (POSIX message queues, System V message queues).

How it actually works

   A SIGNAL is a number, delivered to a process, that INTERRUPTS its
   normal execution to run a handler (either a DEFAULT action the
   kernel provides, or a CUSTOM one your program registered). Common
   ones:

     SIGTERM (15)   "please terminate" -- the POLITE request. A
                    well-behaved program catches this, cleans up
                    (closes files, finishes in-flight work, flushes
                    buffers), and exits on its own terms. This is what
                    `kill <pid>` sends by DEFAULT (no flag needed).
     SIGKILL (9)    "terminate NOW, no negotiation." The kernel
                    forcibly ends the process -- it CANNOT be caught,
                    blocked, or ignored by the program (this is by
                    design: it's the guaranteed last resort). This is
                    what `kill -9 <pid>` sends. No cleanup code runs.
     SIGINT (2)     "interrupt" -- sent when you press Ctrl-C in a
                    terminal. Catchable, like SIGTERM.
     SIGHUP (1)     historically "the terminal hung up"; commonly
                    repurposed by daemons to mean "reload your
                    configuration without restarting."
     SIGSEGV (11)   "you accessed invalid memory" (Chapter 11's
                    segfault) -- delivered BY THE KERNEL when it
                    detects this, not sent deliberately by anyone.
     SIGCHLD (17)   "one of your child processes changed state" (e.g.
                    exited) -- how a parent process is notified without
                    having to constantly poll (Chapter 3's wait()).

   A PROCESS CAN, for most signals: use the DEFAULT action (often
   "terminate," sometimes "ignore," depending on the signal), install a
   CUSTOM HANDLER (run specific code when it arrives), or explicitly
   IGNORE it. SIGKILL and SIGSTOP are the two exceptions -- deliberately
   un-catchable and un-ignorable, so there's always a way to force a
   process to stop, no matter how badly it's misbehaving.

How it actually works

   THE CHAIN, roughly, on a modern PC/server:

     1. FIRMWARE (UEFI, the modern successor to the old BIOS) runs
        FIRST, the instant power is applied -- built into the
        motherboard, independent of any disk. It initialises basic
        hardware, runs a Power-On Self-Test (POST), and looks for a
        BOOTLOADER to hand off to.

     2. THE BOOTLOADER (commonly GRUB on Linux) is a small program,
        itself loaded from disk by the firmware, whose entire job is
        to find and load the actual KERNEL into memory and start it.
        This is why you see a boot MENU (choose a kernel version, boot
        into recovery mode) at this stage -- the bootloader is where
        that choice happens, before the kernel is even running.

     3. THE KERNEL takes over: initialises itself, sets up memory
        management (Chapter 10), detects and initialises hardware
        (loading DEVICE DRIVERS as it goes, Chapter 22), mounts an
        initial, minimal root filesystem, and finally starts the very
        FIRST USER-SPACE PROCESS -- historically always PID 1.

     4. PID 1 (the INIT SYSTEM -- systemd on most modern Linux
        distributions, Chapter 22) takes over from here: it starts every
        other system service (networking, logging, your SSH daemon, your
        application) in the right order, and becomes the ultimate PARENT
        of every process on the machine (Chapter 3) -- if a process's
        original parent dies, PID 1 adopts it.

     5. Eventually, a LOGIN PROMPT (or, for a server, the point where
        SSH becomes reachable) appears -- the machine is now "up."

   EVERY STAGE HANDS OFF TRUST to the next: the firmware trusts the
   bootloader it loads, the bootloader trusts the kernel, the kernel
   trusts PID 1. This CHAIN OF TRUST is exactly what security features
   like UEFI SECURE BOOT are designed to protect (below).

How it actually works

   THE INIT SYSTEM'S JOB: start services (networking, logging, your
   database, your web server) in the correct ORDER (some depend on
   others being up first), keep track of what's supposed to be running,
   RESTART things that crash unexpectedly (if configured to), and clean
   up properly on shutdown.

   SYSTEMD (the default init system on most major Linux distributions
   today -- Debian, Ubuntu, RHEL/Fedora, and others) organises this
   around UNIT FILES: small config files describing a service, what it
   depends on, how to start/stop/restart it, and when it should run.

     $ systemctl status nginx        # is it running? recent log lines?
     $ systemctl start|stop|restart nginx
     $ systemctl enable nginx        # start automatically on boot
     $ journalctl -u nginx           # this service's own logs
                                     # (Chapter 34 covers this properly)

   KERNEL MODULES let the kernel load ADDITIONAL CODE -- typically
   device drivers, but also filesystem implementations and other
   optional functionality -- WITHOUT needing to be entirely recompiled.
   A module is loaded on demand (when the hardware/feature is actually
   needed) or explicitly:

     $ lsmod                    # what's currently loaded
     $ modprobe some_module     # load a module (and its dependencies)
     $ rmmod some_module        # unload one (if nothing's using it)

   A DEVICE DRIVER is the specific kernel code that knows how to talk to
   a particular piece of hardware (a network card, a disk controller, a
   GPU) -- translating the KERNEL's generic notion of "read some bytes
   from this block device" into the exact commands THAT SPECIFIC
   hardware chip understands. This is the layer that makes the rest of
   the OS's abstractions (Chapter 1's "lies") possible at all: the
   kernel doesn't need to know the details of every disk controller ever
   made, because each one's driver handles that translation.

How it actually works

   A FILE DESCRIPTOR (Chapter 2 introduced this) is just a small integer
   a process uses to refer to something it has open. Every process gets
   three, by convention, at startup:
     0   stdin    (standard input)
     1   stdout   (standard output)
     2   stderr   (standard error)
   But a file descriptor can point at FAR more than a regular file:
   a directory, a device (`/dev/sda`, your disk), a pipe (Chapter 19), a
   network socket, or a purely VIRTUAL file the kernel generates on the
   fly.

   THE VIRTUAL FILESYSTEMS ARE THE CLEAREST DEMONSTRATION of this
   philosophy taken all the way:
     /proc    a filesystem that doesn't correspond to anything on disk
             AT ALL -- it's the kernel exposing its OWN internal state
             (running processes, memory stats, CPU info) as if it were a
             directory of files, generated live, on read.
     /sys     similar idea, focused on hardware/device and kernel
             subsystem configuration.
     /dev     special files representing DEVICES -- `/dev/null` (discards
             anything written to it, always returns end-of-file when
             read), `/dev/random` (returns random bytes), `/dev/sda`
             (your actual disk, as a stream of bytes).

   THIS IS NOT JUST A CUTE ABSTRACTION -- it's WHY `ps`, `top`, and
   `free` (Chapter 31) work at all: they don't use some special, secret
   kernel API. They just READ FILES under /proc, the exact same `open`/
   `read`/`close` syscalls (Chapter 2) your own program would use to
   read a text file. You could write your own (much worse) version of
   `ps` in a few lines of shell script, entirely from `/proc`.

How it actually works

   THE CORE DIRECTORIES, and WHY each one exists (this is the part worth
   actually remembering -- the meaning, not just the name):

     /etc      "editable text configuration" -- system-wide config files.
              Nothing here should be a compiled binary or a program's
              own data; it's SETTINGS.
     /var      "variable data" -- files that CHANGE while the system
              runs: logs (`/var/log`), databases, mail queues, caches.
              The opposite of "static."
     /usr      historically "Unix System Resources" (not "user"!) --
              the bulk of installed software: binaries (`/usr/bin`),
              libraries (`/usr/lib`), shared data. On modern systems,
              `/bin` and `/sbin` are often just SYMLINKS into
              `/usr/bin` (see the real-world example).
     /home     personal directories for each human user.
     /root     the root user's OWN home directory -- deliberately
              SEPARATE from `/home`, partly so root's home still exists
              and is usable even if `/home` is a separate, unmounted
              filesystem.
     /tmp      temporary files -- may be CLEARED on every reboot (often
              literally a RAM-backed filesystem, tmpfs) -- never store
              anything here you can't afford to lose.
     /opt      "optional" -- self-contained third-party software that
              doesn't want to scatter itself across `/usr`.
     /dev, /proc, /sys   the virtual filesystems from Chapter 23.
     /boot     the kernel, bootloader config, and other files needed
              extremely early in Chapter 21's boot process.

   THE OVERALL PRINCIPLE: separate STATIC from VARIABLE, separate
   SYSTEM-PROVIDED from USER-INSTALLED from USER-PERSONAL, and put
   things a script can DEPEND ON in a small number of PREDICTABLE
   places. This predictability is the entire point -- it's a decades-old
   social/technical CONVENTION (documented as the "Filesystem Hierarchy
   Standard," maintained by the Linux Foundation), not a technical
   requirement the kernel enforces.

How it actually works

   THE PERMISSION MODEL: `ls -l` shows something like:
     -rwxr-xr--  1  alice  developers  1234  Jan 1 12:00  script.sh
      |||||||||
      |||||||`- OTHERS: read only (r--)
      ||||`----- GROUP: read + execute (r-x)
      `--------- OWNER (alice): read + write + execute (rwx)

   Three PERMISSIONS (read, write, execute), applied to three CATEGORIES
   of user (owner, group, everyone else). For a FILE: read = can view
   contents, write = can modify, execute = can RUN it as a program. For
   a DIRECTORY, the meanings shift slightly: read = can LIST its
   contents, execute = can ENTER it / access files inside by name (even
   without read), write = can CREATE/DELETE entries inside it.

     chmod u+x script.sh      # add execute for the owner (user)
     chmod g-w file.txt       # remove write for the group
     chmod 755 script.sh      # numeric form: owner=7(rwx), group=5(r-x),
                              # other=5(r-x) -- read each digit as a
                              # 3-bit binary sum: r=4, w=2, x=1

   OWNERSHIP: every file has an OWNING USER and an OWNING GROUP
   (`chown`, `chgrp`). This, combined with the permission bits above,
   is the FULL access-control model for a standard Unix file (Chapter
   36 covers more advanced access control beyond this).

   HARD LINKS vs. SYMBOLIC LINKS (both introduced briefly in Chapter
   15, formalised here):
     HARD LINK (`ln target linkname`)    another directory entry
       pointing at the SAME INODE. Indistinguishable from "the
       original" -- there IS no original, just multiple names for one
       piece of data. Can't cross filesystems, can't link a directory.
     SYMBOLIC LINK (`ln -s target linkname`)    a SEPARATE, tiny file
       that just CONTAINS A PATH STRING pointing at the target. Can
       cross filesystems, can link directories, but BREAKS ("dangling
       link") if the target is moved or deleted -- because it's just
       a path, not a direct pointer to the data.

How it actually works

   cp source dest          copy a file
   cp -r sourcedir destdir copy a directory, RECURSIVELY (required for
                          directories -- plain `cp` refuses)
   mv source dest           move OR rename (same command; a rename is
                          just a move within the same directory --
                          Chapter 15's directory-entry model, again)
   rm file                  delete a file. NO confirmation, NO recycle
                          bin, NO undo, by default.
   rm -r directory           delete a directory and everything in it,
                          RECURSIVELY. Combined with -f (force, skip
                          confirmations) this is the single most
                          dangerous common command in daily use.
   mkdir -p a/b/c            create a/b/c, creating any missing parent
                          directories along the way (without -p, this
                          fails if `a` or `a/b` don't already exist)
   touch file                create an empty file, or update an
                          existing file's modification timestamp

Habits that prevent real disasters:

   # Always double-check a wildcard/variable BEFORE running rm with it
   echo rm -rf "$TARGET_DIR"/*      # print first, remove the `echo`
                                   # once you've confirmed it's correct

   # Prefer `rm -i` (interactive, asks per file) while learning
   alias rm='rm -i'                 # add to your shell config

   # For anything precious, copy before you touch it
   cp -a important_dir important_dir.bak

How it actually works

   grep "pattern" file        # print lines matching a pattern
   grep -i "pattern" file      # case-insensitive
   grep -v "pattern" file      # INVERT -- print lines that DON'T match
   grep -r "pattern" dir/      # search recursively through a directory
   grep -c "pattern" file      # just COUNT matching lines
   grep -E "regex|pattern" file # extended regex (alternation, etc.)

   sed 's/old/new/' file       # substitute the FIRST match per line
   sed 's/old/new/g' file      # substitute ALL matches per line (global)
   sed -n '5,10p' file         # print only lines 5 through 10

   awk '{print $1}' file       # print the FIRST whitespace-separated
                               # column of every line
   awk -F, '{print $2}' file    # same, but split on commas instead
   awk '$3 > 100 {print $1}' file  # print column 1, but only for lines
                               # where column 3 is greater than 100

Chained together, this is where the real power shows up:

   # Top 5 most common words in a text file
   cat file.txt | tr ' ' '\n' | sort | uniq -c | sort -rn | head -5

   # Every unique IP address in a web server log, with request counts
   awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

   # Every ERROR line from today's log, with the message only
   grep "ERROR" app.log | awk -F'ERROR: ' '{print $2}'

How it actually works

   command > file        # redirect STDOUT (fd 1) to a file, OVERWRITING it
   command >> file        # same, but APPEND instead of overwrite
   command < file          # redirect STDIN (fd 0) to read FROM a file
   command 2> file         # redirect STDERR (fd 2) to a file
   command > file 2>&1     # redirect stdout to file, THEN point stderr
                          # at "wherever fd 1 currently points" (the
                          # file) -- ORDER MATTERS, this is the classic
                          # gotcha (see below)
   command 2>&1 > file     # THIS DOES SOMETHING DIFFERENT: stderr is
                          # pointed at wherever stdout CURRENTLY goes
                          # (the terminal), THEN stdout is redirected to
                          # the file -- stderr still goes to the
                          # terminal, not the file!
   command1 | command2      # connect command1's STDOUT directly to
                          # command2's STDIN, via a pipe (Chapter 19) --
                          # no file involved at all

Why the order matters, explained properly:

   `2>&1` means "make file descriptor 2 point at wherever file
   descriptor 1 CURRENTLY points, right now, at this moment in the
   command line" -- it's not a persistent link, it's a one-time copy of
   a destination, evaluated left to right.

     > file 2>&1     step 1: fd 1 -> file.  step 2: fd 2 -> (wherever
                     fd 1 now points) -> file. BOTH end up in the file.
     2>&1 > file      step 1: fd 2 -> (wherever fd 1 currently points)
                     -> the terminal (unchanged so far). step 2: fd 1 ->
                     file. fd 2 is STILL pointed at the terminal from
                     step 1 -- it never got updated again.

How it actually works

   find /path -name "*.log"          # search BY NAME, live, recursive
   find /path -type f                # only regular files (-type d = dirs)
   find /path -mtime -1               # MODIFIED in the last 1 day
   find /path -size +100M             # LARGER than 100 megabytes
   find /path -perm -4000             # has the SUID bit set (Chapter 25)
   find /path -name "*.tmp" -delete    # find AND act -- delete matches
                                      # (careful -- test without -delete
                                      # first!)

   locate filename                    # instant, but reads a PRE-BUILT
                                     # INDEX (usually updated once daily
                                     # via a cron job, `updatedb`) --
                                     # can miss recently-created files

   which command                      # shows the FULL PATH of the
                                     # executable that would run, based
                                     # on your $PATH
   type command                        # similar, but also reveals if
                                     # it's a SHELL BUILTIN, alias, or
                                     # function, not just a file on disk

   grep -r "TODO" /path               # search file CONTENTS recursively
                                     # (different from `find`, which
                                     # searches by NAME/metadata, not
                                     # content)

How it actually works

   # Debian/Ubuntu family (dpkg underneath, apt as the friendly frontend)
   sudo apt update                 # refresh the list of AVAILABLE packages
   sudo apt install nginx          # install a package (and its dependencies)
   sudo apt remove nginx           # remove it (config files may remain)
   sudo apt purge nginx            # remove it AND its config
   apt list --installed | grep nginx   # is it installed? which version?

   # RHEL/Fedora family (rpm underneath, dnf -- formerly yum -- as the
   # friendly frontend)
   sudo dnf install nginx
   sudo dnf remove nginx
   dnf list installed | grep nginx

   THE CORE IDEA, either family: a PACKAGE isn't just files -- it's files
   PLUS METADATA, including a list of what it DEPENDS ON (other
   packages, at specific minimum versions). The package manager reads
   this metadata and:
     - refuses to install a package if a dependency can't be satisfied
       (or, more usefully, AUTOMATICALLY installs the needed
       dependencies too)
     - refuses to REMOVE a package if something else still depends on it
       (without an explicit override)
     - tracks exactly what's installed, so it can be queried, upgraded,
       or cleanly removed as a unit, rather than "some files that are
       probably related to nginx, somewhere."

How it actually works

   ps aux                    # every process, classic BSD-style output
   ps -eo pid,ppid,cmd,%cpu,%mem --sort=-%cpu | head   # customised,
                             # sorted by CPU usage -- often more useful
                             # than the default columns
   ps -ef                    # every process, System V-style output
                             # (slightly different columns/conventions)

   top                        # live, auto-refreshing view. Press:
                              #   'q' to quit
                              #   'M' to sort by memory
                              #   'P' to sort by CPU (often the default)
                              #   'k' to kill a process, interactively
   htop                        # a much friendlier, colour, scrollable,
                              # mouse-clickable version of top (not
                              # always pre-installed -- worth adding)

   kill PID                    # send SIGTERM (Chapter 20) -- the polite
                              # request
   kill -9 PID                 # send SIGKILL -- the forceful, un-
                              # ignorable last resort
   kill -l                     # list every signal name/number available
   pkill -f "pattern"           # kill by matching the COMMAND LINE
                              # against a pattern, instead of a PID
   killall processname          # kill every process with this EXACT
                              # name

Reading ps/top columns that confuse people at first:

   %CPU    can exceed 100% on a multi-core machine -- it's the
           percentage of ONE CORE's capacity, so a process using two
           full cores shows ~200%.
   VSZ      virtual memory size (Chapter 10) -- often huge, often
           mostly meaningless on its own.
   RSS      resident set size -- memory ACTUALLY in physical RAM right
           now. Usually the number you actually care about.
   STAT     the process state (Chapter 5: R/S/D/Z/T, plus modifiers
           like `+` for foreground process group, `l` for multi-threaded)

How it actually works

   command &            # run in the BACKGROUND -- your shell keeps
                        # control immediately, doesn't wait
   jobs                  # list your shell's current background/stopped
                        # jobs
   fg %1                  # bring job 1 to the FOREGROUND (wait for it)
   bg %1                   # resume a STOPPED job in the background
   Ctrl-Z                  # SUSPEND (STOP, Chapter 5's `T` state) the
                        # current foreground job
   Ctrl-C                  # send SIGINT to the current foreground job
                        # (Chapter 20)

   nohup command &         # run in the background, AND ignore SIGHUP --
                        # so it survives your terminal/SSH session
                        # ending
   disown %1                # remove a job from your shell's job table
                        # WITHOUT killing it -- it keeps running, but
                        # your shell stops tracking it

Why a background job can still die when you log out: when your SSH session (or terminal) closes, the shell process itself is typically sent SIGHUP ("hang up" — a name dating back to actual telephone modems). By default, many programs' handling of SIGHUP (or simply inheriting the shell's own death) means background jobs started with a plain & can be killed too, unless something protects them — which is exactly nohup's job.

How it actually works

   # /etc/systemd/system/myapp.service -- a minimal, real unit file
   [Unit]
   Description=My Application
   After=network.target              # start AFTER networking is up

   [Service]
   ExecStart=/usr/bin/myapp --config /etc/myapp/config.yml
   Restart=always                    # THE critical line -- restart on
                                     # ANY exit, crash or otherwise
   RestartSec=5                       # wait 5 seconds before restarting
   User=myappuser                     # don't run as root unless you
                                     # genuinely need to

   [Install]
   WantedBy=multi-user.target         # start automatically at normal
                                     # boot

   sudo systemctl daemon-reload        # tell systemd to re-read unit
                                      # files after you edit one
   sudo systemctl enable myapp         # start automatically on boot
   sudo systemctl start myapp           # start it right now
   sudo systemctl status myapp          # is it running? recent log
                                      # lines (Chapter 34)?
   sudo systemctl restart myapp          # stop then start

The Restart= options that matter most:

   no          (the default if unset) never restart automatically
   always       restart no matter HOW it exited -- crash, clean exit,
               anything
   on-failure   restart only if it exited with a non-zero status/signal
               (Chapter 3) -- NOT if it exited cleanly (status 0)

How it actually works

   journalctl                    # ALL journal entries, oldest first
   journalctl -u myapp            # just THIS service's log entries
   journalctl -f                   # FOLLOW -- live tail, like `tail -f`
   journalctl -b                    # just THIS boot session
   journalctl --since "1 hour ago"   # time-filtered
   journalctl -p err                # only ERROR-level and above

   # Traditional plain-text logs still live under /var/log -- not
   # everything goes through the journal (Chapter 24's FHS: /var is
   # "variable data," logs are the classic example)
   tail -f /var/log/syslog          # (Debian/Ubuntu) or
   tail -f /var/log/messages         # (RHEL/Fedora family) -- the
                                    # general system log
   ls /var/log/                      # see what else is being logged --
                                    # often includes per-application
                                    # logs, auth logs, and more

Log rotation (logrotate, the standard tool) exists specifically to prevent logs from growing forever:

   /etc/logrotate.conf and /etc/logrotate.d/*  define, per log file/
   category: how often to rotate (daily/weekly), how many old copies to
   KEEP, whether to compress old copies, and when to DELETE the oldest
   ones entirely. Without this running correctly, a busy service's logs
   genuinely can, and regularly do, grow without bound.

How it actually works

   whoami                    # who am I, right now
   id                         # my user ID, group ID, and all groups
                             # I belong to
   sudo command                # run ONE command as root (by default),
                             # after authenticating (usually your OWN
                             # password, not root's)
   sudo -u alice command        # run as a SPECIFIC other user, not
                             # necessarily root
   su - alice                   # actually SWITCH USER, starting a new
                             # login shell as alice (needs alice's
                             # password, unless you're already root)

   sudo -l                       # list what YOU are allowed to run with
                             # sudo -- often surprisingly restrictive,
                             # by design, on a well-configured system

   /etc/passwd                   # every user account, with basic info
   /etc/group                     # every group, and its members
   /etc/sudoers                    # who can sudo, and to run WHAT
                             # (edit with `visudo`, never directly --
                             # it validates syntax before saving,
                             # preventing you from locking yourself out
                             # with a typo)

Why sudo is preferred over logging in as root directly, as a real, deliberate security design:

   - EVERY sudo invocation is LOGGED (who, what command, when) --
     accountability that a shared root login simply doesn't give you.
   - Access can be SCOPED: a user can be granted sudo rights to run only
     SPECIFIC commands, not unrestricted root access.
   - It requires the USER'S OWN credentials (by default), not a shared
     root password everyone has to know and rotate together.

How it actually works

   LINUX CAPABILITIES split "what root can do" into roughly 40
   independent, individually-grantable PRIVILEGES, instead of the
   old all-or-nothing model (either you're root, with EVERY privilege,
   or you're not, with NONE of them). A few examples:

     CAP_NET_RAW        create raw network sockets (needed by `ping`,
                        packet capture tools)
     CAP_NET_BIND_SERVICE  bind to a port below 1024 (historically
                        root-only)
     CAP_SYS_TIME         change the system clock
     CAP_CHOWN             change file ownership
     CAP_SYS_ADMIN          a large, famously broad grab-bag of
                        administrative operations (genuinely close to
                        "basically root" on its own -- treated with
                        real caution)

   You can grant a SPECIFIC capability to a specific binary, instead of
   making it SUID-root (Chapter 25):

     sudo setcap cap_net_raw+ep /path/to/mytool
     getcap /path/to/mytool          # confirm what was granted

   This means a program with a BUG can, at worst, misuse the ONE
   narrow capability it was granted -- not read every file on the
   system, not create arbitrary users, not do anything else root could
   do. This same mechanism (Linux capabilities) is exactly what
   container runtimes use to run containers with a much smaller,
   scoped set of privileges than genuine root, rather than either "full
   root" or "no privileges at all" (Phase 3 covers this properly).

   ACLs (Access Control Lists) extend the basic owner/group/other model
   to allow permissions for SPECIFIC, ADDITIONAL individual users or
   groups on a single file -- something the basic three-category model
   simply cannot express (e.g. "the owner and the finance group can
   both write to this file, but ALSO grant read-only to this one
   specific contractor's account, without adding them to the finance
   group at all"):

     setfacl -m u:bob:r-- shared_report.csv   # grant bob read-only,
                                              # specifically
     getfacl shared_report.csv                 # see all ACL entries

How it actually works

   ip addr show                # every network INTERFACE and its
                               # assigned IP address(es)
   ip route show                 # the ROUTING TABLE -- where does
                               # traffic to a given destination
                               # actually go?
   cat /etc/resolv.conf           # which DNS server(s) this box asks

   THE CONCEPTS, briefly:
     INTERFACE     a network endpoint the kernel manages -- a physical
                  NIC (`eth0`, `enp0s3`), or a VIRTUAL one (the loopback
                  `lo`, a VPN tunnel, a virtual bridge -- Phase 3's
                  container networking is built entirely on virtual
                  interfaces).
     IP ADDRESS     a numeric address assigned to an interface -- how
                  packets know where to go.
     ROUTE           a rule: "to reach THIS range of addresses, send
                  traffic out THROUGH this interface, via this
                  gateway." Every machine has a DEFAULT route -- where
                  everything NOT otherwise covered goes (usually your
                  router).
     DNS RESOLUTION  translating a hostname ("example.com") into an IP
                  address, by asking a configured DNS server (or
                  checking `/etc/hosts` first, for manual overrides).

   THE LOOPBACK INTERFACE (`lo`, almost always IP address 127.0.0.1,
   "localhost") is worth understanding precisely: it's a VIRTUAL
   interface that loops traffic straight back to the SAME machine,
   entirely within the kernel, never touching real network hardware at
   all. This matters a great deal once you reach containers (Phase 3):
   EACH container typically gets its OWN, separate loopback interface
   and its own "localhost," entirely distinct from the host's -- which
   is exactly why a service listening on `localhost` INSIDE a container
   is invisible from OUTSIDE it, a genuinely common, real point of
   confusion for engineers new to Docker.

How it actually works

   ss -tulpn              # every LISTENING socket on this machine:
                         # protocol (t=tcp, u=udp), listening state,
                         # PID/process name. THE modern replacement for
                         # the older `netstat` (see below).
   ss -tn state established    # currently ESTABLISHED TCP connections

   curl -v https://example.com    # make an HTTP(S) request, -v shows
                                 # the full request/response, headers
                                 # and all
   curl -o file.txt <url>          # save output to a file instead of
                                 # printing it
   curl -I <url>                    # HEAD request only -- just headers,
                                 # fast way to check if something's
                                 # reachable and what it returns

   dig example.com                   # full DNS lookup, with all the
                                 # detail
   dig +short example.com             # just the answer -- the IP(s)
   dig example.com MX                  # a SPECIFIC record type (mail
                                 # servers, in this case)

   tcpdump -i any port 443              # watch raw packets live,
                                 # filtered to a specific port
                                 # (needs root/sudo)

   # basic firewall inspection (varies by distro/tooling -- ufw is
   # common on Ubuntu, firewalld on RHEL/Fedora, both are friendly
   # frontends on top of the kernel's netfilter/iptables/nftables)
   sudo ufw status               # Ubuntu-style
   sudo firewall-cmd --list-all   # RHEL/Fedora-style

How it actually works

   #!/bin/bash              # the SHEBANG -- tells the OS which
                            # interpreter to run this file with
                            # (Chapter 6's exec(), essentially)
   set -e                    # EXIT IMMEDIATELY if any command fails
                            # (non-zero exit status, Chapter 3) --
                            # without this, a script just KEEPS GOING
                            # after a failure, often compounding it
   set -u                    # error out on any UNSET variable, instead
                            # of silently treating it as empty (this is
                            # EXACTLY the class of bug behind Chapter
                            # 26's Steam `rm -rf` incident)
   set -o pipefail            # a pipeline's exit status reflects the
                            # FIRST failing command, not just the last
                            # one -- without this, `false | true` reports
                            # SUCCESS, hiding a real failure upstream

   # a common, recommended combination:
   set -euo pipefail

   VARIABLE="value"             # no spaces around =, by convention
   echo "$VARIABLE"              # ALWAYS quote variable expansions --
                                 # unquoted expansion is a classic
                                 # source of bugs with spaces/globs

   if [ "$1" = "start" ]; then    # test/condition syntax
     echo "starting"
   elif [ "$1" = "stop" ]; then
     echo "stopping"
   else
     echo "usage: $0 start|stop" >&2
     exit 1
   fi

   for file in *.txt; do          # loop over files
     echo "processing $file"
   done

   result=$(some_command)          # capture command output into a
                                  # variable
   if [ $? -ne 0 ]; then           # $? = the PREVIOUS command's exit
                                  # status (Chapter 3) -- but prefer
                                  # `set -e` over manually checking
                                  # this everywhere
     echo "failed" >&2
     exit 1
   fi

How it actually works

   crontab -e                   # edit YOUR OWN scheduled jobs
   crontab -l                    # list them

   # format: minute hour day-of-month month day-of-week   command
   0 2 * * *     /path/to/backup.sh      # every day at 2:00 AM
   */15 * * * *  /path/to/healthcheck.sh  # every 15 minutes
   0 0 1 * *     /path/to/monthly.sh       # midnight, first of every
                                          # month

   ENVIRONMENT VARIABLES are named values available to a process and
   everything it launches -- `$PATH` (where to look for commands),
   `$HOME`, `$USER`, and anything else you or a program has set:

     export MY_VAR="value"        # set it, and make it available to
                                 # any CHILD process this shell launches
     echo $MY_VAR                  # read it
     env                            # list every variable in the CURRENT
                                 # environment
     printenv PATH                  # just one specific variable

   WHY CRON JOBS BEHAVE DIFFERENTLY: cron runs your job with a MINIMAL,
   often DIFFERENT environment than your interactive login shell --
   frequently a much SHORTER `$PATH`, no shell customisations from your
   `.bashrc`, and none of the environment variables your normal
   terminal session has accumulated. A script that calls a command by
   NAME alone (relying on `$PATH` to find it) can work perfectly when
   YOU run it, and fail silently (or with a confusing "command not
   found") under cron, purely because cron's `$PATH` doesn't include
   wherever that command actually lives.

How it actually works

   uptime                    # shows load average as three numbers:
                             # 1-minute, 5-minute, 15-minute averages
   # load average: 2.15, 1.98, 1.75

   WHAT THOSE NUMBERS MEAN: roughly, the average number of processes
   that were EITHER actively running on a CPU OR WAITING to (Chapter 5's
   "R" state) OR -- and this is the widely-misunderstood part, ON LINUX
   SPECIFICALLY -- in UNINTERRUPTIBLE SLEEP (Chapter 5's "D" state,
   almost always waiting on disk/network I/O), averaged over that time
   window. A load average of "4" on a 4-core machine, if it's ALL
   CPU-bound work, means the machine is fully, appropriately busy. The
   SAME number of "4," if it's mostly D-state processes stuck waiting on
   a slow, struggling disk, means something completely different --
   your CPUs might be almost entirely IDLE while load average looks
   identically "high."

   vmstat 1                    # live, per-second system stats. Key
                               # columns:
                               #   r  = processes waiting to RUN (CPU
                               #        pressure)
                               #   b  = processes BLOCKED, in
                               #        uninterruptible sleep (I/O
                               #        pressure -- Chapter 5's D state)
                               #   si/so = swap in/out (Chapter 11's
                               #        memory pressure)
                               #   us/sy/id/wa = %CPU in user time /
                               #        kernel time / idle / waiting on
                               #        I/O

   iostat -x 1                  # per-DEVICE disk I/O detail -- %util
                               # (how busy is this specific disk),
                               # await (average time per I/O request,
                               # in milliseconds -- Chapter 12's
                               # hierarchy, made concrete and
                               # measurable)

How it actually works

   strace -p PID                 # attach to an ALREADY-RUNNING process
                                # and watch its syscalls live (Chapter
                                # 2) -- needs appropriate permissions
   strace command                 # run a NEW command under strace from
                                # the start
   strace -c command                # just a SUMMARY: which syscalls,
                                # how many times, how much time spent
   strace -f command                 # follow child processes too
                                # (Chapter 6's fork/exec)
   strace -e trace=open,read,write command  # filter to SPECIFIC
                                # syscalls, cutting the noise

   lsof -p PID                   # every file/socket/device THIS
                                # process has open
   lsof /path/to/file             # every process with THIS SPECIFIC
                                # file open -- "who's using this?"
   lsof -i :8080                    # what's listening on/using port
                                # 8080
   lsof +L1                         # DELETED files that are still
                                # open (Chapter 15's exact "deleted but
                                # not actually freed" scenario) --
                                # genuinely useful for finding disk
                                # space that `du` can't explain

A practical diagnostic sequence, when a process seems "stuck":

   1. ps -o pid,stat,cmd -p PID    check its STATE (Chapter 5) -- is
                                  it actually running (R), or stuck in
                                  D (uninterruptible I/O wait)?
   2. strace -p PID                 see the LAST syscall it made, and
                                  whether it's making progress at all,
                                  or genuinely blocked on one call
   3. lsof -p PID                    see WHAT it has open -- a file on
                                  a hung network mount? A socket to an
                                  unresponsive service?

How it actually works

   ulimit -a                     # show ALL current limits for this
                                # shell (and anything it launches)
   ulimit -n                      # just open file descriptors
                                # (Chapter 23) -- a VERY commonly hit
                                # limit in real production systems
   ulimit -n 4096                  # raise it (for THIS shell session;
                                # permanent changes need config files --
                                # `/etc/security/limits.conf` on most
                                # systems)
   ulimit -u                        # max number of processes this user
                                # can create -- Chapter 31's fork-bomb
                                # defence, in practice

   CGROUPS (control groups) let the kernel group a set of PROCESSES
   together and apply RESOURCE LIMITS to the GROUP AS A WHOLE -- CPU
   time, memory, I/O bandwidth, process count -- enforced directly by
   the kernel, not just requested politely:

     /sys/fs/cgroup/...    cgroups are exposed as a VIRTUAL FILESYSTEM
                          (Chapter 23's exact philosophy again) -- you
                          create a cgroup by making a directory, and
                          configure it by writing to files inside it.

   THIS IS PRECISELY THE MECHANISM CONTAINERS USE (Phase 3, in full):
   when you run `docker run --memory=512m ...`, Docker is, under the
   hood, creating a cgroup for that container's processes and writing
   `512m` into that cgroup's memory-limit file. If the container's
   processes collectively try to exceed it, the KERNEL'S OOM killer
   (Chapter 13) -- scoped specifically to that cgroup -- kills a process
   inside it, exactly as Chapter 13 described, just now applied to a
   GROUP of processes instead of the whole machine.

How it actually works

   FOUR INGREDIENTS, each one YOU ALREADY KNOW:

     1. PID NAMESPACE      the container's main process becomes PID 1
                          INSIDE its own namespace (Chapter 6's process
                          tree, but the container gets its OWN, private
                          root of that tree) -- it cannot see, and
                          cannot signal (Chapter 20), processes outside
                          its namespace.

     2. NETWORK NAMESPACE   its own network interfaces, IP address, and
                          routing table (Chapter 37) -- including its
                          own, SEPARATE `localhost`, exactly as Chapter
                          37's real-world example previewed.

     3. MOUNT NAMESPACE / ROOT FILESYSTEM   its own view of the
                          filesystem (Chapter 24's hierarchy), typically
                          built from a container IMAGE (Chapter 46) --
                          so it can have its OWN `/etc`, its own
                          installed packages, entirely separate from the
                          host's, even though it's ultimately running on
                          the SAME kernel.

     4. CGROUP             a resource limit (CPU, memory -- Chapters 8,
                          10, 13, 43) applied to the container's
                          process(es) as a group, enforced by the
                          kernel, not politely requested.

   PUT TOGETHER: a container process, from the KERNEL'S perspective, is
   still just an ordinary process -- `ps aux` on the HOST machine shows
   it, right alongside every other process, with its own real PID in the
   host's OWN, top-level PID namespace. It's just been given a
   deliberately RESTRICTED VIEW of the system (can't see other
   namespaces' processes/network) and a CAPPED SLICE of resources
   (cgroups) -- fundamentally the same isolation idea as Chapter 1's
   "the OS as a resource manager" and Chapter 10's virtual memory, just
   composed at a coarser, whole-process-environment level.

How it actually works

   A VIRTUAL MACHINE, via a HYPERVISOR (software like KVM, Xen, or
   VMware that can run directly on hardware, or on top of a host OS):
     - virtualises actual HARDWARE -- CPU, memory, disk, network devices
       are all presented to the guest as if they were real.
     - each VM runs its OWN, COMPLETE, INDEPENDENT KERNEL -- genuinely
       a separate operating system instance, which can even be a
       DIFFERENT OS entirely from the host (a Linux host running a
       Windows VM, for example).
     - ISOLATION is very strong: a kernel bug or crash inside one VM
       cannot directly affect another VM or the host's own kernel,
       because they're not sharing one.
     - COST: a full kernel boot, a larger memory footprint per instance,
       and generally SLOWER startup (often tens of seconds or more) and
       lower density (fewer instances per physical machine, for the same
       hardware).

   A CONTAINER, as Chapter 44 established:
     - shares the HOST's SINGLE kernel -- every container on a machine
       is running on the SAME kernel, at the SAME time.
     - isolation comes from namespaces + cgroups, at the OS level, not
       hardware virtualization.
     - STARTUP is typically near-instant (milliseconds, Chapter 46
       explains why) since there's no kernel to boot -- you're just
       starting a process.
     - DENSITY is far higher for the same hardware -- no per-instance
       kernel memory overhead, much smaller per-instance footprint.
     - the TRADE-OFF: isolation is WEAKER than a VM's -- a sufficiently
       severe kernel vulnerability, or a container-escape technique
       (a real, actively-studied security category, covered in this
       wiki's security guides), can potentially cross between
       containers, or from a container to the host, in a way a
       properly-configured VM's hardware-level boundary generally
       resists more strongly.

   THIS IS WHY "SANDBOXED" CONTAINER RUNTIMES EXIST (gVisor, Kata
   Containers, covered in this wiki's advanced security guide) -- they
   sit deliberately IN BETWEEN: Kata runs each container in its own
   lightweight VM (closer to VM-level isolation, container-like
   ergonomics); gVisor intercepts and re-implements syscalls in
   userspace rather than passing them straight to the host kernel. Real
   engineering teams choose points along this spectrum based on how much
   they trust the WORKLOAD running inside.

How it actually works

   AN IMAGE is a STACK of LAYERS, each one representing a set of
   filesystem CHANGES (files added, modified, or removed) relative to
   the layer below it. Every instruction in a Dockerfile that changes
   the filesystem (`RUN`, `COPY`, `ADD`) typically creates a NEW layer:

     FROM ubuntu:22.04          <- base layers (the OS itself)
     RUN apt-get install -y curl   <- a new layer: curl's files added
     COPY app.py /app/               <- a new layer: your one file added
     CMD ["python3", "/app/app.py"]  <- metadata only, no filesystem
                                     change, no new layer

   THESE LAYERS ARE READ-ONLY and, crucially, CACHEABLE and SHAREABLE:
   if two different images both start `FROM ubuntu:22.04`, they can
   literally SHARE that base layer on disk -- it's stored ONCE, not
   duplicated. When you `docker pull` an image, Docker only downloads
   layers it doesn't ALREADY have locally.

   A UNION FILESYSTEM (commonly OverlayFS on modern Linux) is what
   makes running a container from these layers actually WORK: it
   presents the STACK of read-only layers as if they were a SINGLE,
   unified filesystem, and adds ONE thin, WRITABLE layer on top,
   specific to THIS running container -- any file the container creates
   or modifies while running goes into THAT layer, leaving the
   underlying image layers completely untouched.

   THIS IS EXACTLY WHY CONTAINER STARTUP IS SO FAST: starting a new
   container from an image ALREADY ON DISK doesn't require copying
   gigabytes of filesystem -- it just needs to set up this thin
   writable layer on top of ALREADY-PRESENT, READ-ONLY layers, then
   start the process (Chapter 44). It's a direct, practical consequence
   of Chapter 15's "renaming/linking is cheap, copying data is
   expensive" lesson, applied at the whole-filesystem scale.

How it actually works

   docker pull ubuntu:22.04       # download an IMAGE (Chapter 46) from
                                 # a registry (Chapter 52), if not
                                 # already present locally
   docker run ubuntu:22.04 echo "hi"    # CREATE a container from that
                                 # image, and START the process inside
                                 # it (this implicitly pulls first, if
                                 # needed)
   docker ps                      # list RUNNING containers
   docker ps -a                     # list ALL containers, including
                                 # stopped ones
   docker stop <container>           # send SIGTERM (Chapter 20!),
                                 # wait, then SIGKILL if it hasn't
                                 # exited -- EXACTLY Kubernetes' pod
                                 # termination sequence from Chapter
                                 # 20, one layer down
   docker rm <container>              # remove a STOPPED container
                                 # (and its writable layer, Chapter 46
                                 # -- any unsaved changes are gone)
   docker exec -it <container> bash    # run an ADDITIONAL process
                                 # inside an ALREADY-RUNNING container
                                 # -- useful for poking around, NOT
                                 # how the container's main process
                                 # started
   docker logs <container>              # see the STDOUT/STDERR
                                 # (Chapter 28's file descriptors 1/2)
                                 # of the container's main process

   FLAGS WORTH KNOWING IMMEDIATELY:
     -d          detached -- run in the background, return your
                terminal immediately (Chapter 32's background jobs,
                one layer up)
     -it          interactive + a pseudo-TTY -- lets you actually type
                into the container's process (needed for `docker exec
                ... bash` to feel like a real shell)
     --rm          automatically remove the container once it exits --
                good hygiene for throwaway/test runs
     --name X       give it a memorable name instead of a random one
     -p HOST:CONTAINER   publish a port (Chapter 50 covers this properly)

How it actually works

   # A genuinely good, real-world-shaped example
   FROM python:3.12-slim                 # a SMALL base image, not the
                                        # full, much larger default

   WORKDIR /app

   COPY requirements.txt .                # dependencies FIRST (Chapter
                                        # 46's caching lesson) --
                                        # changes rarely
   RUN pip install --no-cache-dir -r requirements.txt

   COPY . .                                # application code LAST --
                                        # changes often

   RUN useradd --create-home appuser        # create a NON-ROOT user
   USER appuser                              # switch to it -- the
                                        # container's main process now
                                        # runs as appuser, NOT root

   EXPOSE 8000                                # documentation -- doesn't
                                        # actually publish the port
                                        # (Chapter 50 does that)
   CMD ["python3", "app.py"]

The habits that separate a good Dockerfile from a careless one:

   [ ] a SMALL base image (a `-slim`/`-alpine` variant, or a distroless
       image) -- smaller attack surface, faster pulls, less disk.
   [ ] dependencies installed BEFORE application code (Chapter 46).
   [ ] a `.dockerignore` file (works like `.gitignore`) -- excludes
       things like `.git/`, local virtual environments, and secrets
       from ever being COPY-ed into the image at all.
   [ ] a NON-ROOT user for the actual running process.
   [ ] PINNED versions (a specific base image tag, pinned dependency
       versions) rather than `latest`/unpinned -- reproducible builds,
       exactly this wiki's advanced security guide's supply-chain
       chapter's lesson, applied here.
   [ ] MULTI-STAGE builds for compiled languages -- build in one stage
       (with all the compilers/build tools), copy ONLY the final
       artifact into a clean, minimal final stage, so build tooling
       never ships in your production image at all.

How it actually works

   docker volume create mydata          # create a NAMED volume --
                                        # managed by Docker, stored
                                        # OUTSIDE any container's
                                        # writable layer
   docker run -v mydata:/var/lib/postgresql/data postgres  # MOUNT it
                                        # into the container at a
                                        # specific path -- anything the
                                        # container writes there goes
                                        # into the VOLUME, not its
                                        # disposable writable layer

   docker run -v /host/path:/container/path myimage    # a BIND MOUNT --
                                        # maps a SPECIFIC HOST DIRECTORY
                                        # directly into the container
                                        # (useful for local development
                                        # -- edit code on the host, see
                                        # changes immediately inside the
                                        # running container)

   docker volume ls                       # list all named volumes
   docker volume inspect mydata            # where does it actually
                                        # live on the host's disk?
   docker volume rm mydata                  # DELETE it -- and the
                                        # data with it. Deliberately
                                        # separate from removing any
                                        # container.

The mental model that prevents the classic mistake:

   `docker rm <container>`    removes the CONTAINER and its writable
                              layer (Chapter 46). Anything stored ONLY
                              there is GONE.
   `docker volume rm <vol>`    removes the VOLUME, separately, and
                              explicitly. Data in a properly-mounted
                              volume SURVIVES any number of `docker rm`
                              operations on containers that used it --
                              it's a genuinely separate lifecycle,
                              precisely so this can't happen by
                              accident.

How it actually works

   docker network ls                      # Docker creates a default
                                          # "bridge" network; you can
                                          # also make your own

   docker run -p 8080:80 nginx              # PUBLISH: map the HOST's
                                          # port 8080 to the
                                          # CONTAINER's port 80 --
                                          # WITHOUT this flag, the
                                          # container's port is only
                                          # reachable from OTHER
                                          # containers on the same
                                          # Docker network, never from
                                          # the host or outside world

   docker network create mynet               # a custom, USER-DEFINED
                                          # network
   docker run --network mynet --name web nginx
   docker run --network mynet --name db postgres
   # `web` can now reach `db` simply as "db" -- Docker runs a built-in
   # DNS resolver on user-defined networks, resolving CONTAINER NAMES
   # to their internal IP addresses automatically (Chapter 37's DNS
   # concept, provided for you)

   THE NETWORK DRIVERS, briefly:
     bridge (the default)    an isolated virtual network on this host;
                            containers on it can reach each other; the
                            outside world can reach IN only via
                            published ports.
     host                     the container uses the HOST's network
                            namespace DIRECTLY, no isolation at all --
                            faster, but loses the isolation Chapter 44
                            described.
     none                      no networking at all.
     overlay                    spans MULTIPLE HOSTS -- the foundation
                            for Docker Swarm, and conceptually a
                            preview of what Kubernetes' networking
                            model (Part 16) does at a larger scale.

How it actually works

   # docker-compose.yml
   services:
     web:
       build: .                       # build FROM a local Dockerfile
       ports:
         - "8080:80"                   # Chapter 50's -p, declaratively
       depends_on:
         - db                           # start `db` first (doesn't wait
                                       # for it to be actually READY --
                                       # a common, real gotcha)
       environment:
         DATABASE_URL: "postgresql://db:5432/myapp"  # Chapter 50's
                                       # container-name DNS resolution,
                                       # used directly

     db:
       image: postgres:16               # or pull an existing image
       volumes:
         - dbdata:/var/lib/postgresql/data   # Chapter 49's volumes,
                                       # declaratively
       environment:
         POSTGRES_PASSWORD: "devpassword"

   volumes:
     dbdata:                            # declare the named volume

   docker compose up                     # build/pull images, create
                                        # the network, start everything,
                                        # in the RIGHT ORDER
   docker compose up -d                    # detached (background)
   docker compose down                      # stop and remove
                                        # everything (containers,
                                        # network -- volumes SURVIVE by
                                        # default, exactly Chapter 49's
                                        # deliberate separation)
   docker compose logs -f web                # follow ONE service's
                                        # logs
   docker compose ps                          # what's running, in
                                        # this project

How it actually works

   docker pull nginx                  # pulls from Docker Hub by
                                     # default (implicitly:
                                     # docker.io/library/nginx:latest)
   docker pull ghcr.io/someorg/someimage:v1.2.3   # a DIFFERENT
                                     # registry, explicitly -- GitHub
                                     # Container Registry, in this
                                     # example
   docker tag myimage myregistry.example.com/myimage:v1   # tag an
                                     # image for a specific registry
   docker push myregistry.example.com/myimage:v1     # publish it
                                     # (needs `docker login` first)

   PRACTICES WORTH ADOPTING AS DEFAULT:
     [ ] PIN images by a specific TAG (or, more strongly, by DIGEST --
         `nginx@sha256:abc123...`) rather than `latest` -- reproducible,
         and immune to the tag being silently repointed later (this
         wiki's advanced security guide covers exactly this "mutable
         tag" risk in its supply-chain chapters).
     [ ] SCAN images for known vulnerabilities before running them in
         anything beyond a quick local test -- tools like `docker
         scout`, Trivy, or Grype check an image's installed packages
         against known-CVE databases.
     [ ] PREFER official/verified publishers on public registries
         (Docker Hub marks "Official Images" and "Verified Publisher"
         images distinctly) over an unknown, unverified account's image
         claiming to be the same thing.
     [ ] for anything running in production, prefer a PRIVATE registry
         (or a pull-through proxy/mirror) you control, rather than
         pulling directly from the public internet on every deploy.

How it actually works

   RUNNING ONE CONTAINER, on ONE machine, with `docker run`, you already
   fully understand (Part 15). Real production systems need FAR more:

     - MANY containers, across MANY machines (a fleet), because one
       machine's capacity isn't enough, and because a single machine is
       a single point of failure.
     - if a MACHINE DIES, the containers that were running on it need to
       be RESTARTED somewhere else, AUTOMATICALLY -- nobody wants a
       human paged at 3am to manually run `docker run` on a replacement
       server.
     - containers need to FIND EACH OTHER reliably, even as individual
       instances come and go, are rescheduled, or scale up and down
       (Chapter 56).
     - the whole fleet needs to SCALE -- more instances of a busy
       service during high traffic, fewer during quiet periods
       (Chapter 58) -- again, ideally without a human manually
       intervening every time.
     - deploying a NEW VERSION of an application across many running
       instances needs to happen WITHOUT DOWNTIME, and be safely
       REVERSIBLE if something's wrong (Chapter 60).

   KUBERNETES is, fundamentally, a system that takes a DECLARATIVE
   DESCRIPTION of what you want ("I want 5 copies of this container
   running, with these resource limits, exposed on this port") and
   CONTINUOUSLY, AUTOMATICALLY works to make reality match that
   description -- restarting failed instances, rescheduling them onto
   healthy machines, routing traffic to whichever instances are
   currently healthy -- all without a human re-issuing the request
   every time something changes. This "reconcile actual state toward
   desired state, continuously, forever" idea is the single most
   important concept to carry into every chapter that follows.

How it actually works

   THE CONTROL PLANE (the "brain," typically running on dedicated
   control-plane nodes):
     API SERVER          the front door -- EVERYTHING (kubectl, every
                        other control-plane component, every kubelet)
                        talks to Kubernetes ONLY through this REST API.
                        Nothing bypasses it.
     etcd                  a distributed, consistent key-value store --
                        the SOURCE OF TRUTH for the ENTIRE cluster's
                        state (every object's desired configuration,
                        current status).
     SCHEDULER              watches for newly-created pods with no node
                        assigned yet, and DECIDES which node they should
                        run on (based on resource availability,
                        constraints -- Chapter 58 covers this properly).
     CONTROLLER MANAGER      runs the actual "reconcile actual state
                        toward desired state" LOOPS (Chapter 53's core
                        idea) -- a Deployment controller noticing "I
                        should have 5 replicas but only 3 are running"
                        and creating 2 more, for example.

   THE NODE (the "muscle," where your actual containers run -- a
   cluster typically has many of these):
     KUBELET                 the agent running on EVERY node -- talks
                        to the API server, and makes sure the
                        containers THIS node has been assigned are
                        actually running, restarting them locally if
                        they crash (Chapter 33's systemd `Restart=`
                        idea, one layer up).
     CONTAINER RUNTIME         the actual thing running containers on
                        this node -- `containerd` (very commonly, these
                        days) or another compatible runtime -- doing
                        the literal Chapter 44/47 work.
     KUBE-PROXY                handles the networking rules that make
                        Services (Chapter 56) actually route traffic to
                        the right pods on this node.

   THE ARCHITECTURAL PAYOFF: because the CONTROL PLANE decides "what
   should run" and NODES independently, locally, keep "what's actually
   running" matching their OWN last-known assignment (via the kubelet),
   a TEMPORARY control-plane disruption does NOT immediately stop
   already-scheduled, already-running pods from continuing to run and
   serve traffic on healthy nodes -- the kubelet keeps enforcing the
   last instruction it received, even if it can't currently reach the
   API server for a NEW one. This graceful-degradation property is a
   deliberate, well-documented design characteristic, not an accident.

How it actually works

   # a minimal, real Deployment manifest
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: web-app
   spec:
     replicas: 3                     # I want THREE identical pods
     selector:
       matchLabels:
         app: web-app
     template:                        # the POD TEMPLATE -- what each
                                     # replica actually looks like
       metadata:
         labels:
           app: web-app
       spec:
         containers:
         - name: web
           image: nginx:1.25
           ports:
           - containerPort: 80

   THE THREE LAYERS, and why each exists:
     POD          one or more containers that share a network namespace
                (Chapter 44) and are always scheduled TOGETHER, on the
                SAME node. Pods are treated as EPHEMERAL and DISPOSABLE
                -- you almost never create one directly in real use;
                you describe what you want at the Deployment level and
                let Kubernetes create/destroy Pods as needed.
     REPLICASET    ensures a SPECIFIED NUMBER of pods matching a
                selector are running, AT ALL TIMES -- if one crashes or
                is deleted, the ReplicaSet's controller (Chapter 54)
                notices and creates a replacement.
     DEPLOYMENT     manages ReplicaSets OVER TIME, enabling ROLLING
                UPDATES (Chapter 60): when you change the pod template
                (a new image version, say), the Deployment creates a
                NEW ReplicaSet with the new template, gradually scales
                it UP while scaling the OLD ReplicaSet DOWN, and keeps
                the old ReplicaSet around (scaled to zero) so a ROLLBACK
                is fast if needed.

   kubectl apply -f deployment.yaml     # create or UPDATE, declaratively
   kubectl get deployments
   kubectl get replicasets               # you'll see one per version
                                        # you've ever deployed (kept
                                        # around, scaled to 0, for
                                        # rollback -- up to a
                                        # configurable history limit)
   kubectl get pods                       # the actual, currently
                                        # running containers
   kubectl scale deployment web-app --replicas=5   # imperatively change
                                        # the desired count

How it actually works

   apiVersion: v1
   kind: Service
   metadata:
     name: web-app-svc
   spec:
     selector:
       app: web-app              # matches the SAME label from Chapter
                                # 55's Deployment's pod template
     ports:
     - port: 80                   # the Service's OWN port
       targetPort: 80              # the port on the PODS to send
                                # traffic to
     type: ClusterIP               # the default -- internal-only
                                # (Chapter 37's networking, one layer
                                # up)

   THE CORE MECHANISM: a Service doesn't point at specific PODS by
   name or IP -- it uses a LABEL SELECTOR, continuously, live. Whatever
   pods CURRENTLY have matching labels are automatically included as
   valid destinations -- as pods are created, destroyed, or rescheduled
   (each getting a NEW IP, Chapter 55), the Service's routing updates
   automatically, with no manual reconfiguration ever needed. This is
   EXACTLY Docker Compose's container-name DNS resolution (Chapter 50),
   but far more dynamic -- resolving to a live, continuously-updated SET
   of healthy pods, not one fixed container.

   SERVICE TYPES, briefly:
     ClusterIP (default)   reachable only FROM WITHIN the cluster --
                          the normal way services talk to EACH OTHER.
     NodePort                 additionally exposes the service on a
                          specific port on EVERY node's own IP --
                          simple, but rarely the final answer for real
                          external traffic.
     LoadBalancer              (on a cloud provider) provisions an
                          actual, external cloud load balancer pointing
                          at the service -- the standard way to expose
                          something to the real internet.

   EVERY SERVICE also gets a DNS NAME, automatically, resolvable from
   anywhere in the cluster: `web-app-svc.default.svc.cluster.local`
   (or just `web-app-svc` from within the same namespace) -- Chapter
   37's DNS resolution, provided automatically by Kubernetes' own
   internal DNS (CoreDNS, typically).

How it actually works

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: app-config
   data:
     LOG_LEVEL: "info"
     API_TIMEOUT: "30s"

   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: app-secret
   type: Opaque
   stringData:                      # plaintext HERE, in the manifest --
     DB_PASSWORD: "supersecret123"    # Kubernetes base64-ENCODES it on
                                    # storage (NOT encryption, Chapter
                                    # 6 of the advanced security guide
                                    # in this wiki covers this
                                    # precisely)

   # using both in a pod spec
   spec:
     containers:
     - name: app
       image: myapp:1.0
       envFrom:
       - configMapRef:
           name: app-config           # every key becomes an env var
       env:
       - name: DB_PASSWORD
         valueFrom:
           secretKeyRef:
             name: app-secret
             key: DB_PASSWORD

   kubectl create configmap app-config --from-literal=LOG_LEVEL=info
   kubectl get configmap app-config -o yaml
   kubectl create secret generic app-secret --from-literal=DB_PASSWORD=supersecret123
   kubectl get secret app-secret -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
      # ^ trivially decoded -- this is exactly the point: it's NOT
      #   encryption, it's encoding, exactly G1's Chapter 6 lesson
      #   ("Base64 is an envelope, not a safe"), applied directly to
      #   Kubernetes Secrets

How it actually works

   # resource REQUESTS and LIMITS -- the scheduler's primary inputs
   resources:
     requests:                # "I need AT LEAST this much" -- the
       cpu: "250m"              # SCHEDULER uses this to decide which
       memory: "256Mi"           # node has room
     limits:                   # "never let me use MORE than this" --
       cpu: "500m"                # enforced by the KERNEL via cgroups
       memory: "512Mi"             # (Chapter 43!) -- exceed the memory
                                 # limit and the OOM killer (Chapter 13)
                                 # acts, scoped to this pod's cgroup,
                                 # exactly as Chapter 44 described

   THE SCHEDULER'S JOB, each time a new pod needs a node: look at every
   node's AVAILABLE capacity (total minus what's already requested by
   other pods there), filter out nodes that don't meet any
   CONSTRAINTS (taints/tolerations, node selectors, affinity rules --
   ways to say "only run on nodes with this label" or "never run two of
   these on the same node"), and pick a node from what's left (using a
   scoring algorithm to balance the cluster reasonably).

   IF NO NODE HAS ENOUGH ROOM: the pod stays in `Pending` state
   (Chapter 59 covers diagnosing this) -- it's not an error exactly,
   it's the scheduler correctly reporting "I have nowhere to safely put
   this yet."

   AUTOSCALING, two distinct, complementary layers:
     HORIZONTAL POD AUTOSCALER (HPA)   watches a metric (commonly CPU
       or memory utilisation, or a custom metric) and automatically
       adjusts a Deployment's REPLICA COUNT within a range you define --
       more pods when busy, fewer when quiet. This is Chapter 55's
       ReplicaSet mechanism, driven automatically instead of manually.
     CLUSTER AUTOSCALER (on a cloud provider)   watches for pods stuck
       `Pending` because NO EXISTING NODE has room, and automatically
       provisions ADDITIONAL NODES (real cloud instances) to make room --
       then, just as importantly, removes nodes again once they're no
       longer needed, to control cost.
   TOGETHER: HPA scales PODS to match demand; the cluster autoscaler
   scales the underlying MACHINES to make sure there's actually room for
   however many pods HPA has decided are needed.

How it actually works

The debugging order that works, almost every time:

   1. kubectl get pods                 -- what STATUS is it in?
   2. kubectl describe pod <name>      -- read the Events at the bottom
   3. kubectl logs <name>              -- what did the app itself say?
   4. kubectl logs <name> --previous   -- what did it say before it
                                          last crashed? (for CrashLoopBackOff)
   5. kubectl get events --sort-by=.metadata.creationTimestamp
                                       -- cluster-wide event timeline

The six buckets, by STATUS column in kubectl get pods:

   Pending
     The Pod has not been scheduled to a node yet. Cause is almost
     always in `describe`'s Events: "Insufficient cpu", "Insufficient
     memory", "0/5 nodes are available: 5 node(s) had taint
     {node-role: master} that the pod didn't tolerate", or a
     PersistentVolumeClaim that has no matching PersistentVolume.
     Fix: right-size requests, add tolerations, or fix storage.

   ImagePullBackOff / ErrImagePull
     Kubernetes cannot pull the container image. Almost always a typo
     in the image name/tag, a private registry with no imagePullSecret
     configured, or (surprisingly common) the image tag was deleted or
     overwritten by a later build.
     Fix: `kubectl describe pod` shows the exact registry error.

   CrashLoopBackOff
     The container starts, then exits, repeatedly, with increasing
     backoff delay between restarts. The container image and pull are
     fine -- the *application* is crashing. This is the one where
     `kubectl logs --previous` is essential, because by the time you
     look, the current attempt's logs may be empty (crashed before
     printing anything) while the previous attempt's logs show the
     real stack trace.
     Common causes: missing environment variable / Secret, cannot
     reach a database on startup, misconfigured config file, the
     entrypoint script has a typo.

   Pending forever with "0/N nodes are available: N Insufficient memory"
     This is a scheduling problem, not a crash. Either lower the
     Pod's memory *request*, add nodes, or find and evict a
     resource-hogging Pod that's wasting reserved-but-unused capacity.

   OOMKilled (seen in `kubectl describe pod`, "Last State: Terminated,
   Reason: OOMKilled")
     The container exceeded its memory *limit* and the kernel's OOM
     killer (Chapter 11) killed it. Fix: raise the limit if the
     workload legitimately needs more memory, or fix a memory leak if
     it doesn't.

   Running, but Service traffic still fails (0/1 Ready)
     The container process is alive but the readiness probe is
     failing -- e.g. the app takes 45 seconds to warm up but the
     probe's initialDelaySeconds is 5. Kubernetes is correctly
     refusing to send traffic to a Pod that says it isn't ready;
     the fix is almost always in the probe configuration.

How it actually works

   ROLLING UPDATE, Deployment with replicas: 4,
   maxSurge: 1, maxUnavailable: 1 (the defaults, roughly):

   Start:   [v1] [v1] [v1] [v1]                4 old, 0 new
   Step 1:  [v1] [v1] [v1] [v1] [v2]            surge: create 1 new
                                    ^ wait for readiness probe to pass
   Step 2:  [v1] [v1] [v1]      [v2] [v2]      1 old terminated, 1 more new
   Step 3:  [v1] [v1]          [v2] [v2] [v2]
   Step 4:  [v1]               [v2] [v2] [v2] [v2]
   Step 5:                     [v2] [v2] [v2] [v2]   done

   At every step, the Service only ever sends traffic to Pods that
   are Ready. A new Pod that fails its readiness probe simply never
   receives traffic and the rollout STALLS there -- it does not
   proceed to kill more old Pods. This is the core safety property.

Two different probes, often confused:

   livenessProbe:   "is this process healthy, or should it be
                      killed and restarted?" A failing liveness
                      probe causes a container restart. Wrong
                      liveness config -> death spiral: probe fails
                      under normal load -> container restarted ->
                      cold-starts and is slow -> fails probe again.

   readinessProbe:  "should this Pod currently receive traffic?"
                      A failing readiness probe removes the Pod from
                      the Service's endpoint list WITHOUT killing it.
                      This is the one that gates rollouts and is safe
                      to make strict, because failing it doesn't
                      destroy anything -- it just stops sending new
                      requests until the Pod says it's ready again.

Rollback is a first-class operation, not a manual redeploy:

kubectl rollout history deployment/api
kubectl rollout undo deployment/api              # back to previous revision
kubectl rollout undo deployment/api --to-revision=3
kubectl rollout status deployment/api            # watch a rollout live

How it actually works

   A CHART is a directory:

     mychart/
       Chart.yaml          -- name, version, description
       values.yaml         -- default configuration values
       templates/
         deployment.yaml   -- Go-template YAML, referencing .Values.*
         service.yaml
         configmap.yaml
       charts/              -- bundled dependencies (sub-charts)

   templates/deployment.yaml might contain:

     replicas: {{ .Values.replicaCount }}
     image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

   values.yaml (the defaults):
     replicaCount: 1
     image:
       repository: myapp
       tag: latest

   A production override, values-prod.yaml:
     replicaCount: 5
     image:
       tag: v2.4.1

The commands that matter day to day:

helm install myapp ./mychart -f values-prod.yaml   # first install
helm upgrade myapp ./mychart -f values-prod.yaml   # apply changes
helm rollback myapp 3                              # back to revision 3
helm history myapp                                 # see revisions
helm template ./mychart -f values-prod.yaml         # render YAML,
                                                     # don't apply --
                                                     # use this to review
                                                     # exactly what
                                                     # would be sent to
                                                     # the cluster
helm uninstall myapp

Every helm upgrade is a new numbered revision, tracked by Helm as a Secret in the cluster -- which is what makes helm rollback possible: it isn't magic, it's "re-apply the exact rendered YAML from revision N."

How it actually works

The project, concretely:

   STEP 1 -- Write a program (any language) that:
     a) reads a large text file (a few hundred MB -- e.g. concatenate
        some log files, or download a public dataset)
     b) counts word frequency (a classic, CPU + I/O mixed workload)
     c) writes the top 20 words to stdout

   STEP 2 -- Measure it plainly first:
     /usr/bin/time -v ./wordcount bigfile.txt
     Read every line of the output: elapsed time, user time, system
     time, maximum resident set size, major/minor page faults,
     voluntary/involuntary context switches. Write down what each
     number means in your own words, using Chapters 4-13's vocabulary.

   STEP 3 -- Trace its syscalls:
     strace -c ./wordcount bigfile.txt
     Which syscall dominates the count? Which dominates total time?
     (Usually `read` calls dominate count; for a naive line-by-line
     reader, this can be enormous -- tens of thousands of small reads
     instead of a handful of large ones.)

   STEP 4 -- Change the I/O strategy and re-measure:
     Rewrite the file-reading part to read in large chunks (e.g. 1MB
     buffers) instead of line-by-line with an unbuffered reader, or
     use memory-mapping (mmap) instead of read(). Re-run STEP 2 and
     STEP 3. Compare: syscall count, elapsed time, major page faults.

   STEP 5 -- Constrain it with cgroups and watch it suffer honestly:
     systemd-run --scope -p MemoryMax=50M ./wordcount hugefile.txt
     If the file is bigger than 50MB and your program loads it all
     into memory at once, watch it get OOM-killed. Check `dmesg` or
     `journalctl -k` for the OOM killer's log line (Chapter 11).
     Then fix the program to stream the file instead of loading it
     all at once, and show it now completes under the same limit.

How it actually works

   STEP 1 -- Pick or write a small app with a real dependency:
     A web API (any language) that reads/writes to a Postgres or Redis
     instance is ideal -- it forces you to deal with networking between
     containers, not just a single standalone process.

   STEP 2 -- Write a Dockerfile that gets every practice right:
     - Multi-stage build: a "builder" stage with full build tools,
       a final stage with only the runtime and the built artifact
     - Pinned, minimal base image (e.g. python:3.12-slim, not
       python:latest)
     - A dedicated, non-root user (USER appuser)
     - A .dockerignore excluding .git, node_modules/venv, secrets
     - A HEALTHCHECK instruction that hits a real readiness endpoint
     - Layers ordered so dependency installation is cached separately
       from application code copying (Chapter 48)

   STEP 3 -- Write docker-compose.yml with two services:
     the app, and its dependency (postgres or redis), using
     depends_on, a named volume for the database's data directory,
     and environment variables (not hardcoded secrets) for connection
     details.

   STEP 4 -- Verify the security properties, don't just assume them:
     docker inspect <container> --format '{{.Config.User}}'
       -- confirm it is NOT empty/root
     docker exec <container> whoami
       -- confirm the actual running user
     docker history <your-image>
       -- confirm no secrets baked into a layer
     docker images <your-image> --format '{{.Size}}'
       -- compare against a naive single-stage build's size

   STEP 5 -- Break the dependency on purpose and observe:
     docker compose stop postgres
     -- does your app crash hard, retry gracefully, or hang forever?
     Fix it to fail predictably (clear error, healthcheck reflects it)
     rather than hanging or crash-looping without useful logs.

How it actually works

   STEP 1 -- Set up a local cluster:
     minikube start        (or: kind create cluster)
     kubectl get nodes     -- confirm it's up

   STEP 2 -- Write the Kubernetes manifests (or a Helm chart, per
   Chapter 61, if you want the extra practice):
     - a Deployment for your app image, with:
         resources.requests and resources.limits set deliberately,
         not left blank
         a livenessProbe hitting a cheap endpoint
         a readinessProbe hitting an endpoint that actually checks
         the database connection
     - a ConfigMap for non-secret configuration
     - a Secret for the database password / connection string
     - a Service (ClusterIP is fine for this exercise) in front of
       the Deployment
     - if using a database, a separate Deployment (or StatefulSet) +
       Service + PersistentVolumeClaim for it

   STEP 3 -- Deploy and verify with the Chapter 59 debugging sequence,
   even if nothing is actually broken -- practice the habit:
     kubectl apply -f .
     kubectl get pods -w
     kubectl describe pod <name>
     kubectl logs <name>

   STEP 4 -- Perform a real rolling update:
     Change something visible (a response header, a log line) in your
     app, rebuild the image with a new tag, kubectl set image ..., and
     watch `kubectl rollout status` complete cleanly.

   STEP 5 -- Perform a rollback, on purpose, under a manufactured
   failure:
     Deploy a broken version (bad image tag, or a readiness probe
     pointed at a path that 404s) and confirm the rollout correctly
     STALLS rather than replacing every old Pod. Then:
     kubectl rollout undo deployment/<name>
     and confirm service was never actually interrupted for end users
     during either the bad deploy attempt or the rollback.

How it actually works

   SETUP (ask a friend, a study partner, or do this to your own
   deployment a week later after you've forgotten the details):
   Introduce exactly ONE of these planted failures, chosen at random,
   into the Project 3 Kubernetes deployment:

     a) Change the readiness probe's path to one that 404s
     b) Set a memory limit lower than the app actually needs, causing
        OOMKilled restarts under load
     c) Change a ConfigMap value the app needs (e.g. a database host
        name) to something wrong
     d) Change the Service's label selector so it no longer matches
        the Deployment's Pods
     e) Set an overly aggressive livenessProbe (very short
        initialDelaySeconds) causing restart-loop on a normally-slow-
        starting app
     f) Fill the node's disk (or a PVC) to force a storage-related
        failure

   YOUR PROCESS, once told only "the app is down, users are getting
   errors":

     1. DETECT   -- confirm the actual user-visible symptom first
                    (curl the Service / Ingress yourself)
     2. DIAGNOSE -- run the Chapter 59 sequence: get pods, describe,
                    logs, events. Form a hypothesis before changing
                    anything.
     3. MITIGATE -- stop the bleeding first. Often this means
                    `kubectl rollout undo` rather than root-causing
                    live, if a previous known-good revision exists.
     4. FIX      -- once mitigated, identify and fix the actual root
                    cause (not just the symptom).
     5. WRITE    -- a half-page postmortem: what broke, how you
                    detected it, what you tried that didn't help,
                    what actually fixed it, and one concrete change
                    (a monitor, an alert, a stricter code review
                    check) that would have caught this earlier.

How it actually works

Think in layers:

   IMAGE / BASELINE     distro version, kernel, packages, hardening profile
   BOOT CONFIG          kernel cmdline, initramfs, systemd units, cloud-init
   RUNTIME STATE        running processes, open files, mounts, sockets, cgroups
   DECLARED CONFIG      Ansible/Chef/Puppet/Nix/Terraform/Kubernetes manifests
   OBSERVED BEHAVIOUR   metrics, logs, traces, alerts, user symptoms

When the layers disagree, you have drift. Drift is not automatically bad — an emergency one-line fix is sometimes the right move — but untracked drift is how the same incident returns weeks later.

How it actually works

Useful commands for real diagnosis:

systemctl status myapp.service
journalctl -u myapp.service -b --no-pager
systemctl cat myapp.service
systemd-analyze verify ./myapp.service
systemd-analyze critical-chain myapp.service
systemctl show myapp.service -p Restart -p User -p Group -p ExecStart

Useful unit-file ideas:

[Unit]
Description=Example API
After=network-online.target
Wants=network-online.target

[Service]
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server
Restart=on-failure
RestartSec=5s
TimeoutStartSec=30s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp /var/log/myapp

[Install]
WantedBy=multi-user.target

Do not cargo-cult hardening directives. ProtectSystem=strict is excellent when the write paths are explicit; it is an outage when the service legitimately needs to write somewhere you forgot to list.

How it actually works

Common stack:

   physical disk / cloud volume
      -> partition table
      -> LUKS encryption, maybe
      -> RAID or cloud-level replication, maybe
      -> LVM physical volume / volume group / logical volume, maybe
      -> filesystem: ext4, XFS, Btrfs, ZFS, etc.
      -> mount options
      -> application write pattern and fsync behaviour

Safe read-only inventory commands:

lsblk -f
findmnt
df -hT
du -xh /var 2>/dev/null | sort -h | tail
sudo journalctl -p warning -b --no-pager | grep -Ei 'disk|ext4|xfs|nvme|i/o'

Growth workflow, conceptually:

   enlarge underlying disk/volume
   grow partition or PV
   grow LVM logical volume, if used
   grow filesystem
   verify with df/findmnt/application metrics

The exact commands differ by filesystem and cloud provider. Do not paste a resize recipe into production unless you have confirmed the device names, stack layout, filesystem type, and backup state.

How it actually works

Core inspection sequence:

ip addr
ip route
ip rule
ss -ltnp
ss -s
resolvectl status 2>/dev/null || cat /etc/resolv.conf
sudo nft list ruleset 2>/dev/null || sudo iptables -S

Packet-path mental model:

   process socket
      -> network namespace
      -> local routing decision
      -> firewall/conntrack/NAT hooks
      -> interface queue
      -> driver/NIC or virtual bridge/veth
      -> remote network

Container networking is the same model with extra namespaces and virtual interfaces. A Kubernetes Pod has its own network namespace; Services add load balancing and NAT/eBPF rules depending on the cluster implementation.

How it actually works

Use a ladder:

   1. User symptom: slow endpoint, timeout, error rate, cost spike
   2. Golden signals: latency, traffic, errors, saturation
   3. Host view: CPU, memory, disk, network, run queue
   4. Process view: threads, file descriptors, syscalls, allocations
   5. Code/path view: profiles, flame graphs, traces
   6. Kernel view: scheduler, block I/O, TCP, locks, eBPF probes

Examples:

uptime
vmstat 1 5
pidstat -durh 1 5 2>/dev/null || true
perf top 2>/dev/null

perf answers "where is CPU time going?" eBPF-based tools can answer questions like "which process is opening this file?", "which TCP connects are slow?", or "which kernel function is hot?" without permanently instrumenting the application. OpenTelemetry answers a different question: how requests move through distributed services.

How it actually works

Hardening layers:

   identity        dedicated user/group, no shared admin account
   filesystem      ownership, mode bits, mount options, read-only paths
   privilege       sudoers scope, Linux capabilities, no_new_privs
   isolation       systemd sandboxing, namespaces, containers, chroot where useful
   MAC             SELinux or AppArmor policy
   kernel          seccomp, module loading rules, exploit mitigations
   audit           auditd/journald/SIEM events tied to real questions

Useful inspection:

id myapp 2>/dev/null || true
getcap -r /usr/bin /usr/sbin 2>/dev/null
sudo -l
aa-status 2>/dev/null || true
getenforce 2>/dev/null || true
systemctl show myapp.service -p NoNewPrivileges -p CapabilityBoundingSet 2>/dev/null

How it actually works

Production-safe incident rhythm:

   1. State the user-visible impact.
   2. Freeze context: what changed, when, on which hosts?
   3. Capture volatile evidence before restarting: logs, process state,
      sockets, open files, resource usage.
   4. Mitigate using the smallest reversible action.
   5. Verify recovery from the user's perspective.
   6. Root-cause with evidence.
   7. Refactor: tests, alerts, runbooks, config, capacity, rollback path.

Evidence commands:

date -Is
uptime
free -h
df -hT
ss -s
systemctl --failed
journalctl -p warning -b --no-pager | tail -100

Refactoring examples:

   repeated disk-full incident
      -> add log rotation, quota/retention, filesystem alert, runbook

   repeated OOM kill
      -> measure working set, fix leak or limit, add memory dashboard,
         tune request/limit, add load test

   repeated manual service restart
      -> fix readiness check, dependency ordering, backoff, crash alert,
         and deploy rollback

How it actually works

The scheduler has three kinds of object (G, M, P, covered in the Go guide's scheduler chapter):

   G  goroutine     your code; thousands to millions
   M  machine       an OS thread (what `ps -L` and /proc/<pid>/status "Threads:" count)
   P  processor     a slot allowed to run Go code; there are GOMAXPROCS of them

How a goroutine waits determines what happens to its thread:

strace -f -c ./yourprogram on any Go binary shows the runtime's own system calls: futex (threads sleeping and waking), epoll_pwait (the netpoller), clone (new threads), mmap (heap growth), rt_sigprocmask (signal handling).

How it actually works

Running a program well with os/exec:

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "ffmpeg", "-i", in, out)  // killed if ctx expires
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}     // own process group...
cmd.Cancel = func() error {                               // ...so we can kill ALL of it,
	return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)  // including grandchildren
}
out, err := cmd.CombinedOutput()
var ee *exec.ExitError
if errors.As(err, &ee) {
	fmt.Println("exit code", ee.ExitCode())               // -1 means "killed by a signal"
}

The Setpgid + negative-PID kill matters because killing only the direct child leaves its children (a shell's pipeline, say) running as orphans.

PID 1 is special. When a process exits, its parent must wait() for it; until then it's a zombie (Chapter 5): no memory, but a PID-table entry. When a parent dies first, the kernel re-parents its children to PID 1. A normal init (systemd, tini) loops on wait() to reap them. A Go server running as PID 1 in a container doesn't, unless you write that loop.

Also: the kernel doesn't apply default signal actions to PID 1, so a PID 1 without a SIGTERM handler simply ignores docker stop and is SIGKILLed after the grace period. Go installs handlers when you call signal.Notify.

How it actually works

cpu.max = 150000 100000          quota 150 ms per 100 ms period (= 1.5 CPUs)

GOMAXPROCS=8, all busy:
  |########--------------------------------| 8 threads x ~19 ms = 150 ms used by t=19ms
  |        ^ throttled for the remaining ~81 ms of every period
GOMAXPROCS=2, all busy:
  |####################################----| 2 threads share the quota evenly; little or no stall

Go 1.25 made GOMAXPROCS container-aware: on Linux it reads the cgroup CPU limit, rounds it up, applies a minimum of 2, and re-checks periodically in case the limit changes. Measured with Go 1.26: --cpus=0.5, 1, and 1.5 all gave GOMAXPROCS = 2, and --cpus=2.2 gave 3. The GOMAXPROCS environment variable still overrides it.

How it actually works

• GOGC=100 (the default): run the next GC when the heap has grown 100% beyond the live heap after the last GC. Live 160 MB means the next GC at about 320 MB.
• GOMEMLIMIT (Go 1.19+): a soft limit on the runtime's total memory. As memory approaches it, the GC runs more often, whatever GOGC says.
• The cgroup limit (memory.max): a hard limit enforced by the kernel. On reaching it, the kernel reclaims what it can and then OOM-kills.

Set GOMEMLIMIT to roughly 90% of the container's memory limit, leaving room for non-heap memory: goroutine stacks, runtime metadata, and C libraries in cgo programs.

How it actually works

os.WriteFile(path, data)       -> bytes in the PAGE CACHE; on disk "eventually" (seconds later)
f.Sync()   (fsync)             -> this file's data AND metadata on stable storage
os.Rename(tmp, path)           -> atomic: any reader sees the old file or the new one, never a mix
dir.Sync() (fsync on the dir)  -> the rename itself (a directory entry) is durable

• Why not just O_TRUNC and write? Between the truncate and the last write there's a window in which the file is empty or partial, and a crash inside that window leaves it that way.
• Why the same directory? rename(2) is only atomic within one filesystem. Across filesystems it's a copy.
• Why fsync the directory? The new name lives in the directory's data. Until that's flushed, a crash can bring back the old name, or no name at all.

How it actually works

• Two limits: the soft limit is what's enforced, and the hard limit is the ceiling an unprivileged process can raise its soft limit to.
• Go raises its own soft limit at startup to the hard limit minus one (since Go 1.19; the Go source explains that the −1 lets Go notice if another process later changes its limit with prlimit). Go then restores the original soft limit for any program it starts with os/exec, so old software that breaks above 1,024 descriptors isn't affected.
• epoll: the runtime registers every network descriptor with one epoll instance. A dedicated wait (epoll_pwait) wakes exactly the goroutines whose sockets are ready. That's why Chapter 73 measured 10,000 socket-blocked goroutines on 16 threads.

How it actually works

CPU% requires two samples: (ticks_after − ticks_before) / USER_HZ / interval. USER_HZ is 100 on practically every Linux system.

How it actually works

 run (parent, on the host)
  ├─ mkdir /sys/fs/cgroup/minicontainer-<pid>; write pids.max, memory.max
  └─ re-exec /proc/self/exe "child" with:
       Cloneflags: NEWUTS | NEWPID | NEWNS | NEWIPC | NEWNET
       CgroupFD:   the new cgroup              (Go 1.20+: born inside it, no race)
 child (PID 1 in its own PID namespace)
  ├─ sethostname("minicontainer")
  ├─ chroot(rootfs); chdir("/")
  ├─ mount proc on /proc                      (so ps sees only this namespace)
  └─ run the user's command

Real runtimes (runc, crun) add pivot_root instead of chroot (which can be escaped by a root process), user namespaces, seccomp filters, capability dropping, a network setup (veth pair + bridge, as in the TCP/IP guide's Chapter 52), and OCI image unpacking.

How it actually works

perf (Chapter 70) works on Go binaries too. Go keeps frame pointers on amd64 and arm64, so perf record -g gets proper call stacks, and it can show time spent in the kernel, which pprof can't.

How it actually works

Honest gaps, organized by phase, with a concrete next resource for each:

   PHASE 1 (OS concepts) -- what this guide under-covered:

     - Kernel internals beyond the syscall boundary (how the scheduler,
       memory manager, and VFS are actually implemented in C).
       Next: "Linux Kernel Development" by Robert Love; read the
       actual kernel source for a subsystem you care about.

     - Filesystem internals in depth (how ext4's journal actually
       works, B-trees in Btrfs/XFS, copy-on-write).
       Next: "Operating Systems: Three Easy Pieces" (free online,
       ostep.org) -- covers this more rigorously than this guide did.

     - Real-time operating systems and embedded constraints.
       Not covered here at all -- a genuinely different field with
       different tradeoffs (predictability over throughput).

   PHASE 2 (Linux fundamentals) -- what this guide under-covered:

     - Advanced networking beyond host-level diagnosis: BGP, EVPN/VXLAN,
       traffic engineering, load-balancer internals, and routing at scale.
       Ch 69 gave you the Linux packet-path model; running a network team is
       a deeper specialization.
       Next: "TCP/IP Illustrated" by W. Richard Stevens; the
       Linux Advanced Routing & Traffic Control HOWTO (lartc.org).

     - Security hardening at dedicated security-engineering depth:
       SELinux/AppArmor policy authoring, kernel exploit mitigation research,
       audit frameworks, incident forensics, and compliance operations. Ch 71
       gave you the Linux operator's model; security is its own field.
       Next: the [[real-life-security-guide-v1]] companion guide in
       this wiki's security/ folder covers this ground in depth.

     - Distributed systems fundamentals (consensus, CAP theorem,
       replication) -- necessary for understanding *why* etcd and
       Kubernetes's control plane are built the way they are, only
       gestured at in Chapter 54.
       Next: "Designing Data-Intensive Applications" by Martin
       Kleppmann.

   PHASE 3 (Containers/Kubernetes) -- what this guide under-covered:

     - Kubernetes networking internals (CNI plugins, how Services
       actually route packets via iptables/IPVS, service meshes like
       Istio/Linkerd).
       Next: "Kubernetes Networking" chapters in "Kubernetes in
       Action" by Marko Luksa; the CNCF's Istio documentation.

     - Multi-cluster and multi-region operation, GitOps (ArgoCD/Flux),
       and progressive delivery (canary releases, feature flags at
       the infrastructure layer).
       Next: CNCF's own "Cloud Native Landscape" as a map of what
       exists; ArgoCD's official documentation as a hands-on start.

     - Observability platform engineering: Prometheus architecture and PromQL
       at scale, distributed tracing storage, high-cardinality control, log
       pipeline economics, and profiling as a service. Ch 70 gave you the
       troubleshooting ladder; operating the observability platform is a
       specialty.
       Next: "Distributed Systems Observability" by Cindy Sridharan
       (free online).

     - Cost and capacity planning at scale -- right-sizing a fleet of
       nodes, spot/preemptible instance strategy, multi-tenant
       cluster isolation.
       Next: your cloud provider's own well-architected / cost
       optimization documentation (AWS, GCP, and Azure each publish
       one).

A study plan, if you want one rather than just a map: pick ONE gap above that's closest to your actual job's needs, go deep on it for a month using the named resource, and come back to this guide's capstones (Part 18) using what you learned as an added constraint (e.g. "redo Project 3, but also add a NetworkPolicy" once you've studied Kubernetes networking). Breadth-first learning (a little of everything) is how you got here; depth-first learning (one thing at a time, thoroughly) is how you become genuinely senior at any one of them.

How it actually works

A short, durable list of places to check periodically, organized by how often they change:

   RARELY CHANGES (concepts -- revisit if you feel rusty, not on a
   schedule):
     - The core OS concepts in Parts 1-6 of this guide. Processes,
       memory, syscalls, and the boot sequence have been stable in
       their fundamentals for decades and will likely remain so.

   CHANGES OVER YEARS (check every year or two):
     - Linux kernel release notes (kernelnewbies.org/LinuxChanges) --
       skimming major version summaries tells you what's newly
       possible or newly deprecated.
     - Your distribution's release notes (Ubuntu LTS, Debian stable,
       RHEL) if you operate a specific distribution professionally.

   CHANGES OVER MONTHS (check a few times a year, or when planning
   an upgrade):
     - Kubernetes release notes and the deprecated API migration
       guide (kubernetes.io/docs/reference/using-api/deprecation-guide)
       -- checking this before cluster upgrades is a professional habit;
       skipping it is a common way to break production.
     - Docker Engine release notes, if you build images as part of
       your job.
     - The CNCF's annual "Cloud Native Landscape" survey, for a sense
       of which tools in the ecosystem are gaining or losing adoption.

   CONTINUOUS (follow loosely, don't obsess):
     - Your own company's or a major cloud provider's incident
       postmortems and engineering blog -- reading how other
       real teams break things and fix them is, chapter for chapter,
       some of the most useful continuing education available, and
       it's exactly the kind of material this guide's "Real-world
       example" sections were built from.
     - CVE feeds relevant to your stack (e.g. a mailing list or RSS
       feed for the specific Linux distribution, container runtime,
       and Kubernetes distribution you actually run).