Appendix A -- Glossary (plain language)
Terms are defined the way they're actually used in conversation, not the way a spec defines them. Where a chapter covers a term in depth, it's named so you can go back.
- ACL (Access Control List) -- an extra, finer-grained permission list on a file, beyond owner/group/other. Ch36.
- Bind mount -- a Docker mount that points at a specific path on the host filesystem, as opposed to a Docker-managed named volume. Ch49.
- Boot loader -- the small program (e.g. GRUB) that loads the kernel into memory and hands it control. Ch21.
- Capability (Linux) -- a narrow privilege carved out of root, such as binding low ports, changing ownership, or loading kernel modules. Ch71.
- cgroup (control group) -- a kernel mechanism that limits and accounts for a group of processes' use of CPU, memory, and I/O. The basis of container resource limits. Ch43, Ch44.
- ConfigMap -- a Kubernetes object holding non-secret configuration data, injected into Pods as environment variables or files. Ch57.
- Container -- a process (or group of processes) running with its own PID/network/mount namespaces and a cgroup limit, giving the illusion of an isolated machine while sharing the host kernel. Ch44.
- Context switch -- the CPU saving one process/thread's state and loading another's, so it can run something else. Ch9.
- Control plane -- the set of Kubernetes components (API server, etcd, scheduler, controller manager) that decide what should be running, as opposed to the nodes that actually run it. Ch54.
- CVE -- Common Vulnerabilities and Exposures, the standard public identifier for a specific named security vulnerability.
- Daemon -- a background process with no controlling terminal,
typically started at boot and running for the system's lifetime
(e.g.
sshd,dockerd). Ch22. - Deployment -- a Kubernetes object describing a desired Pod template and replica count; manages ReplicaSets, which manage Pods. Ch55.
- Drift -- the difference between intended configuration and actual runtime state across one host or a fleet. Ch66.
- eBPF -- a Linux kernel technology for safely running small programs attached to kernel/application events, widely used for observability, networking, and security tooling. Ch70.
- etcd -- the distributed key-value store that is Kubernetes' single source of truth for cluster state. Ch54.
- exec() -- the syscall family that replaces a process's running program with a different one, keeping the same PID. Ch6.
- fork() -- the syscall that creates a new process by duplicating the calling process. Ch6.
- FHS (Filesystem Hierarchy Standard) -- the convention that defines
what
/etc,/var,/usr,/tmp, etc. are each for. Ch24. - HPA (Horizontal Pod Autoscaler) -- a Kubernetes controller that changes a Deployment's replica count based on observed metrics like CPU usage. Ch58.
- Image (container) -- a read-only, layered filesystem snapshot plus metadata that a container is started from. Ch46.
- Inode -- the on-disk data structure holding a file's metadata (permissions, size, pointers to data blocks) and identity, separate from its name. Ch15.
- IPC (Inter-Process Communication) -- any mechanism (pipes, sockets, shared memory, message queues) letting separate processes exchange data. Ch19.
- Journaling (filesystem) -- writing a log of intended changes before applying them, so a crash mid-write can be recovered from without a full filesystem scan. Ch15.
- kubelet -- the agent running on every Kubernetes node that talks to the container runtime and reports Pod status back to the control plane. Ch54.
- Layer (image) -- one immutable filesystem diff stacked inside a container image; layers are cached and shared across images. Ch46.
- Liveness probe -- a Kubernetes health check whose failure causes the container to be restarted. Ch60.
- Load average -- a rolling measure of the number of processes running or waiting for a resource (CPU or, on Linux, also I/O), averaged over 1/5/15 minutes. Ch41.
- LVM (Logical Volume Manager) -- a Linux storage layer that lets you group physical storage and carve it into logical volumes that can often be resized more flexibly than raw partitions. Ch68.
- MAC (Mandatory Access Control) -- policy-based access control, such as SELinux or AppArmor, enforced beyond ordinary Unix mode bits. Ch71.
- Namespace (kernel) -- a kernel mechanism that gives a process its own view of a global resource (PIDs, network interfaces, mounts, hostname) separate from the rest of the system. Ch44. (Not to be confused with a Kubernetes namespace, which is a logical grouping of objects inside one cluster.)
- OOM killer -- the kernel's Out-Of-Memory killer, which forcibly kills a process to free memory when the system is critically low. Ch13.
- Page cache -- RAM the kernel uses to cache recently read/written disk data, transparently, so re-reads don't touch the disk. Ch16.
- Page fault -- a CPU trap raised when a virtual memory access can't be immediately satisfied; "minor" is resolved in RAM, "major" requires disk I/O. Ch11.
- PID 1 -- the first process started by the kernel after boot
(traditionally
init, now usuallysystemd); becomes the ultimate parent/reaper of every other process. Ch21. - Pod -- the smallest deployable unit in Kubernetes: one or more containers that share a network namespace and are always scheduled together. Ch55.
- Readiness probe -- a Kubernetes health check whose failure removes a Pod from a Service's traffic without restarting it. Ch60.
- ReplicaSet -- a Kubernetes controller ensuring a specified number of identical Pod replicas are running; usually managed for you by a Deployment. Ch55.
- Rolling update -- replacing old Pods with new ones a few at a time, gated by readiness, so the workload stays available throughout. Ch60.
- Secret (Kubernetes) -- like a ConfigMap, but intended for sensitive values; base64-encoded, not encrypted, by default. Ch57.
- Service (Kubernetes) -- a stable virtual IP and DNS name that load balances traffic across a dynamic set of Pods, chosen by label selector. Ch56.
- Signal -- an asynchronous notification sent to a process by the kernel or another process (e.g. SIGTERM, SIGKILL, SIGHUP). Ch20.
- SUID (Set User ID) bit -- a permission bit that makes a program run with its owner's privileges rather than the invoking user's. Ch25.
- Syscall (system call) -- the only sanctioned way a user-space program asks the kernel to do something privileged. Ch2.
- systemd unit -- a declarative object managed by systemd: service, timer, socket, mount, target, and more. Ch33, Ch67.
- TLB (Translation Lookaside Buffer) -- a small, fast, on-CPU cache of recent virtual-to-physical address translations. Ch11.
- Virtual memory -- the abstraction giving every process its own private, contiguous-looking address space, actually backed by scattered physical RAM and disk via the page table. Ch10.
- Volume (Docker/Kubernetes) -- storage that persists independently of a container's lifecycle. Ch49.
- Zombie process -- a process that has exited but whose exit status
hasn't yet been collected (
wait()ed on) by its parent. Ch5.
Appendix B -- Command cheat sheet
Grouped by task, not alphabetically -- so you can find "the command for what I'm trying to do" rather than needing to already know its name.
PROCESSES
ps aux list all processes
ps -ef --forest list with a parent/child tree
top / htop live process viewer
kill -TERM <pid> ask a process to exit gracefully
kill -KILL <pid> (or kill -9) force-kill, unconditionally
pkill -f <pattern> kill by matching command line
nohup ./long_task.sh & survive terminal disconnect
jobs / fg / bg manage foreground/background jobs
strace -p <pid> trace a running process's syscalls
lsof -p <pid> list a process's open files/sockets
FILES AND TEXT
ls -la list files, including permissions
cp -r src/ dst/ copy a directory recursively
find . -name "*.log" -mtime -1 files named *.log, modified <1 day ago
grep -rn "TODO" . recursive, line-numbered search
sed 's/foo/bar/g' file replace foo with bar, all occurrences
awk '{print $1}' file print the first whitespace-delimited field
tail -f /var/log/app.log follow a growing log file live
chmod 640 file owner rw, group r, others none
chown user:group file change ownership
PACKAGES AND SERVICES
sudo apt install <pkg> / sudo dnf install <pkg> install a package
sudo apt remove <pkg> / sudo apt purge <pkg> remove (purge also drops config)
systemctl status <svc> is it running, and why/why not
systemctl restart <svc> restart a service now
systemctl enable <svc> make it start automatically at boot
journalctl -u <svc> -f follow a service's live logs
NETWORKING
ip addr list network interfaces and IPs
ss -tlnp list listening TCP sockets and owners
curl -v https://host/path make an HTTP request, verbosely
dig +short example.com resolve a hostname, short output
tcpdump -i eth0 port 443 capture packets on a port
PERFORMANCE
uptime load average (1/5/15 min)
vmstat 1 CPU/memory/swap snapshot every second
iostat -x 1 per-disk I/O stats every second
free -h memory and swap usage
df -h / du -sh * disk space by filesystem / by directory
ulimit -n current open-file limit
DOCKER
docker build -t name:tag . build an image from a Dockerfile
docker run -d --name x image run a container, detached
docker ps -a list containers, including stopped
docker logs -f <name> follow a container's logs
docker exec -it <name> sh open a shell inside a running container
docker compose up -d start every service in compose.yaml
docker system df show Docker disk usage before cleanup
docker system prune remove unused Docker data after review
KUBERNETES
kubectl get pods -o wide list pods with node/IP info
kubectl describe pod <name> full detail, including Events
kubectl logs <name> --previous logs from the last crashed instance
kubectl apply -f file.yaml create/update from a manifest
kubectl rollout status deploy/x watch a rollout to completion
kubectl rollout undo deploy/x roll back to the previous revision
kubectl exec -it <pod> -- sh shell inside a running pod
helm install/upgrade/rollback install, update, or revert a chart
ADVANCED OPERATIONS
uname -a kernel/version fingerprint
cat /etc/os-release distro fingerprint
systemctl cat <svc> effective unit and overrides
systemd-analyze critical-chain boot/service dependency timing
lsblk -f block devices and filesystems
findmnt mounted filesystems and options
journalctl --disk-usage journald storage use
ip route / ip rule routing tables and policy routing
nft list ruleset nftables firewall rules
ip netns list network namespaces
perf stat <cmd> hardware/software performance counters
getcap -r /usr/bin /usr/sbin files with Linux capabilities
aa-status / getenforce AppArmor / SELinux mode, if installed
Appendix C -- Troubleshooting playbook ("the server is slow")
A flowchart-style sequence for the single most common vague complaint in this field. Work top to bottom; each step either finds the cause or rules out a whole category.
"The server / service is slow." Start here.
1. IS IT ACTUALLY THE SERVER, OR THE NETWORK / CLIENT?
curl -w "@curl-format.txt" the endpoint yourself, from somewhere
close to the server. If IT is fast but users report slow, suspect
DNS, a CDN, or a client-side/network path issue instead. (Ch37, Ch38)
2. CHECK THE FOUR BASELINE RESOURCES, IN THIS ORDER:
a) CPU uptime / vmstat 1
Load average far above core count + high 'us'/'sy'
time -> CPU-bound. (Ch41)
b) MEMORY free -h ; check "available", not "free"
Available near zero, swap climbing, or dmesg showing
"Out of memory: Killed process" -> memory pressure.
(Ch13, Ch16)
c) DISK I/O iostat -x 1 ; check %util and await
%util near 100 with high await -> disk-bound.
df -h AND df -i -- full data blocks or full inodes
both cause writes to fail. (Ch14, Ch15, Ch16)
d) PROCESSES WAITING ON I/O vmstat 1, the 'b' column
Consistently non-zero -> processes blocked on I/O,
not the CPU scheduler. (Ch41)
3. IF ALL FOUR LOOK FINE, SUSPECT THE APPLICATION ITSELF:
strace -c -p <pid> which syscalls dominate?
lsof -p <pid> how many files/sockets open --
approaching the ulimit? (Ch43)
Check application-level logs and metrics (slow queries, GC pauses,
thread pool exhaustion) -- this is now outside OS-level diagnosis.
4. IF IT'S A CONTAINER / POD SPECIFICALLY:
docker stats <name> or kubectl top pod <name>
hitting its memory/CPU LIMIT? -> OOMKilled or CPU-throttled.
(Ch43, Ch58)
kubectl describe pod <name> -- any restarts, and what reason?
kubectl logs <name> --previous -- if it has restarted at all.
(Ch59)
5. IF IT JUST STARTED BEING SLOW AFTER A CHANGE:
What deployed, and when? (kubectl rollout history / your CI log)
Mitigate first: roll back if a known-good previous version exists,
BEFORE continuing to root-cause live. (Ch65)
6. ONCE MITIGATED: write down what you checked, in what order, and
what you found at each step -- whether or not it turned out to be
the cause. This is the raw material for the postmortem (Ch65) and
for improving this exact playbook next time.
Appendix D -- Further reading, and how this guide's facts were verified
Every "Real-world example" in this guide draws on one of three kinds of source, chosen deliberately over inventing a plausible-sounding but made-up scenario:
1. NAMED, PUBLIC INCIDENTS with a primary source -- a postmortem,
a CVE record, a court filing, or reporting from a named
publication at the time (e.g. Knight Capital's SEC filing, the
Mars Pathfinder JPL technical report, CVE-2016-5195 "Dirty COW,"
CVE-2021-3156 "Baron Samedit," the Therac-25 case studies by
Leveson and Turner).
2. OFFICIAL PROJECT DOCUMENTATION AND SOURCE HISTORY -- the
Kubernetes docs, Docker's own engineering blog, the Linux kernel's
own changelogs and LKML archives, systemd's design documents.
3. WIDELY CORROBORATED INDUSTRY PATTERNS -- practices and failure
modes (like the "healthy-looking crash" readiness-probe problem,
or the naive-line-by-line-I/O performance trap) that are described
independently, in similar terms, by multiple unrelated
engineering organizations' public writing (Google's SRE book,
Netflix's tech blog, Cloudflare's, Monzo's, and others) -- treated
as reliable precisely because they were arrived at independently.
Where a fact's precision matters (a version number, a date, a specific
percentage), this guide favors the vaguer-but-correct phrasing
("around 2021-2022," "roughly," "a widely cited figure") over a
precise-sounding number it could not verify. If you plan to cite any
of these examples yourself -- in a blog post, a talk, or an interview
answer -- look up the named primary source above and read it directly;
secondhand retellings, including this guide's, drift from the original
facts over time.
Broad, durable resources worth owning, beyond the per-chapter "Further reading" lists:
- "Operating Systems: Three Easy Pieces," Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau -- free at ostep.org
- W. Richard Stevens and Stephen A. Rago, "Advanced Programming in the UNIX Environment"
- Brendan Gregg, "Systems Performance: Enterprise and the Cloud," 2nd edition
- Google, "Site Reliability Engineering" and "The Site Reliability Workbook" -- both free at sre.google/books
- The official Kubernetes documentation at kubernetes.io/docs -- kept current, unlike any book, by design
- Julia Evans' zines and blog (jvns.ca) -- exceptionally clear, practically-minded writing on Linux internals and debugging
Appendix E -- Answers to "Check yourself"
Answers are deliberately brief -- a correct direction and the key fact, not a full re-explanation. If an answer doesn't make sense, that's a signal to reread the chapter's "How it actually works" section, not a sign you should have memorized it.
CHAPTER 1
1. (a) each program gets its own private memory -- really shared RAM
partitioned by virtual memory; (b) each program appears to have the
CPU to itself -- really time-sliced by the scheduler; (c) files/
devices look uniform -- really very different hardware behind a
common syscall interface.
2. Without an OS, any program could read/overwrite any other program's
memory or monopolize the CPU/devices -- there was no enforced
isolation.
3. Isolation and crash containment (one tab crashing doesn't take down
the browser or other tabs) -- paid for with higher total memory
usage, since less is shared between processes.
CHAPTER 2
1. The CPU itself, via a hardware-enforced privilege level (ring 0 vs.
ring 3 on x86); user-mode code physically cannot execute privileged
instructions.
2. E.g. read() (read bytes from a fd), write() (write bytes), fork()
(create a process), open()/close() (manage file descriptors),
mmap() (map memory).
3. Because they require a trap into kernel mode -- a context switch in
privilege level, with real CPU overhead -- unlike an ordinary
function call that stays in user space.
4. It was a race condition in the memory subsystem letting an
unprivileged user write to files they should only be able to read,
and it had existed in the kernel for roughly nine years before
discovery -- dangerous because of both severity (arbitrary write)
and how long it went unnoticed.
CHAPTER 3
1. Shell forks a child, the child execs the target program, the kernel
loads and runs it, the program's stdout is inherited from the shell,
the shell wait()s for the exit code, and the shell prints its next
prompt.
2. fork() creates a near-identical copy of the calling process; exec()
replaces the current process's program with a different one,
keeping the same PID.
3. A zombie is a process that exited but whose exit status hasn't been
collected yet by wait() -- it exists so the parent has a chance to
read the exit code before the kernel discards it.
4. If the pipeline's last command succeeds even though an earlier
command failed (without `pipefail`), the shell's exit code reflects
only the last command -- masking the real failure.
CHAPTER 4
1. E.g. PID, memory mappings, open file descriptors, signal handlers/
pending signals, scheduling priority, current working directory.
2. Creating a process means the kernel allocating and initializing a
full set of these structures (and, without copy-on-write
optimizations, copying memory) -- far more work than a function
call, which just pushes a stack frame.
3. Handling many concurrent connections without the memory and
context-switch overhead of one thread (or process) per connection.
CHAPTER 5
1. Running (on CPU), Ready/Runnable (waiting for CPU), Blocked/
Uninterruptible (D state, waiting on I/O), Stopped (suspended),
Zombie (exited, awaiting reap).
2. D state (uninterruptible sleep) means the process is waiting on a
kernel-level I/O operation that ignores signals, including SIGKILL,
until that I/O completes or times out.
3. Check its process state with `ps` or `top` (is it D-state, blocked
on I/O?) before assuming it's a CPU or application logic problem.
CHAPTER 6
1. fork() duplicates the calling process; exec() replaces the current
program image. They're separate because you often want to do
setup (redirect fds, change environment) in the child AFTER
forking but BEFORE the new program starts.
2. In the parent, fork() returns the child's PID; in the child, it
returns 0.
3. Forking a process per connection gives strong isolation (a crashed
connection handler can't corrupt another's memory) at the cost of
higher per-connection overhead than threads, which share memory
but risk one bug corrupting all connections.
CHAPTER 7
1. Threads share the same address space, open file descriptors, and
signal handlers; each thread keeps its own stack, registers, and
program counter.
2. No full memory-space duplication is needed -- just a new stack and
kernel bookkeeping, versus copying/mapping an entire address space
for a new process.
3. To avoid the complexity and bugs of shared-memory concurrency
(locks, race conditions) for I/O-bound work, using a single-threaded
event loop plus non-blocking I/O instead.
CHAPTER 8
1. Deciding which of many runnable processes/threads gets the CPU next,
and for how long, so the system feels responsive and fair.
2. CFS models an idealized "perfectly fair" CPU that gives every
runnable task an equal share, and picks the task with the least
accumulated virtual runtime so far to run next.
3. No -- it makes the process MORE LIKELY to be scheduled sooner and
get a larger share of CPU time, but the kernel can still delay it
if higher-priority (lower nice) work is also runnable.
CHAPTER 9
1. An interrupt is a hardware or software signal that stops the CPU's
current work to run a handler immediately; a context switch is the
act of saving one task's state and loading another's -- interrupts
are one common trigger for a context switch.
2. Cache and TLB contents built up for the old task are cold for the
new one, causing slower memory access after the switch; and the
scheduler itself consumes CPU cycles deciding what to run next.
3. Because it's driven by I/O readiness events rather than one
OS thread per connection, so far fewer threads exist to be switched
between in the first place.
CHAPTER 10
1. Giving each process its own private, contiguous-looking address
space, so processes can't see or corrupt each other's memory and
don't need to know about physical RAM layout.
2. The Memory Management Unit (MMU), using the page table (with the
TLB as its cache) that the kernel maintains per process.
3. ASLR randomizes where a process's memory regions are placed in ITS
virtual address space on each run -- only possible because that
address space is a kernel-managed abstraction, not physical
addresses.
CHAPTER 11
1. A minor fault is resolved by mapping already-resident RAM (fast); a
major fault requires reading the page in from disk/swap (orders of
magnitude slower).
2. The TLB caches recent virtual-to-physical address translations, so
the CPU doesn't have to walk the page table on every memory access.
3. Thrashing is when the system spends most of its time swapping pages
in and out rather than doing real work; CPU usage can look low
because processes are mostly blocked waiting on disk I/O, not
actually computing.
CHAPTER 12
1. CPU registers, CPU cache (L1/L2/L3), RAM, SSD/NVMe, spinning disk /
network storage -- fastest and smallest to slowest and largest.
2. Roughly five to six orders of magnitude (RAM access ~100ns, SSD
~10-100 microseconds, spinning disk several milliseconds).
3. Because the same underlying trade-off -- a small fast layer in front
of a large slow one, exploiting locality of access -- recurs at
every scale of computing, from hardware to distributed systems.
CHAPTER 13
1. The kernel allows more memory to be "promised" via allocation calls
than physically exists, betting that not all processes will use
their full allocation simultaneously -- it does this to allow
normal patterns like large sparse allocations and fork()'s
copy-on-write.
2. It kills a process to free memory when the system is critically low
and can't satisfy an allocation; it picks a victim using a
badness/oom_score heuristic weighted heavily by memory usage.
3. The OOM killer's heuristic favors killing whichever process is using
the MOST memory at that moment, which can be the database rather
than a smaller process that is actually leaking, unless scores are
deliberately adjusted.
4. Kubernetes sets a Pod's memory `limit` as its cgroup's hard cap --
exceeding it triggers the same kernel OOM mechanism, reported back
as the Pod status "OOMKilled."
CHAPTER 14
1. Program's write() -> VFS -> filesystem driver -> page cache -> block
layer/I/O scheduler -> device driver -> physical disk.
2. No -- write() returning success usually only means the data reached
the page cache. fsync() (or O_DIRECT/O_SYNC) is what guarantees it's
physically durable on disk.
3. That "successfully written" and "durably persisted" are different
guarantees, and applications that assumed the former implied the
latter lost data when the assumption broke.
CHAPTER 15
1. The inode (metadata/identity), the data blocks (actual content), and
the directory entry / filename (a pointer to the inode).
2. The inode and its data blocks aren't freed until the last reference
to them -- including any process still holding the file open -- is
gone, so an open-but-unlinked file keeps consuming space.
3. Because inodes are a separate, finite resource from data blocks --
if all inodes are used (e.g. by millions of tiny files), no new file
can be created even with free space, since there's nowhere to record
its metadata.
4. Journaling protects against a crash mid-write leaving the filesystem
structure itself inconsistent, by logging intended metadata changes
before applying them; recovery then just replays the journal instead
of scanning the entire disk for inconsistencies.
CHAPTER 16
1. RAM the kernel uses to cache recently accessed disk data; it uses
otherwise-idle RAM because unused RAM provides zero benefit, while
cached data can turn a disk read into a much faster RAM read.
2. "Free" is genuinely unused RAM; "available" also includes page-cache
memory that can be reclaimed instantly if a process needs it --
available is the number that reflects real memory pressure.
3. Spinning disks have real mechanical seek latency, so scheduling
(e.g. elevator algorithms) reorders requests to minimize head
movement; SSDs have no seek penalty, so schedulers instead focus on
maximizing parallel queue depth.
CHAPTER 17
1. An operation is not atomic if it can be interrupted partway through
by another thread/process seeing or changing shared state -- this
matters because the interruption can leave data in a state neither
party intended.
2. Because races depend on precise timing that may only manifest under
specific load, thread interleavings, or hardware -- tests that don't
reproduce that exact timing pass every time until production
conditions trigger it.
3. The bug depended on a specific, rare sequence of rapid keystrokes
within a race window that testers were unlikely to reproduce, and
the software had no hardware interlock to catch the resulting
over-radiation dose.
CHAPTER 18
1. A mutex guarantees mutual exclusion -- only one thread holds it at a
time, and typically only the holder can release it; a semaphore
maintains a count and allows up to N holders, and can be signaled by
a different thread than the one that waited on it.
2. Mutual exclusion, hold-and-wait, no preemption, circular wait; break
the cycle by enforcing a consistent lock-acquisition order across
all threads.
3. A low-priority task held a mutex needed by a high-priority task,
while medium-priority tasks kept preempting the low-priority one --
priority inversion; fixed with priority inheritance, temporarily
boosting the lock-holder's priority.
CHAPTER 19
1. E.g. pipes (simple, one-directional, related processes), sockets
(bidirectional, works across machines, more overhead), shared memory
(fastest, but no built-in synchronization), message queues
(structured messages, kernel-managed delivery).
2. A Unix domain socket stays entirely in the kernel without going
through the full TCP/IP network stack (no packetization, checksums,
or loopback network device), so it has lower overhead.
3. Anyone who can write to the Docker socket can effectively run
arbitrary commands as root on the host via the Docker daemon --
it's equivalent to root access.
CHAPTER 20
1. SIGTERM asks a process to terminate gracefully and can be caught/
handled; SIGKILL forces immediate termination and is delivered
directly by the kernel, never reaching the process's own code, so it
cannot be caught, blocked, or ignored.
2. SIGTERM.
3. If it ignores SIGTERM, Kubernetes will wait out the grace period and
then send SIGKILL, which doesn't give the app a chance to finish
in-flight requests or close connections cleanly -- causing dropped
requests during deploys.
CHAPTER 21
1. Power on -> firmware (UEFI/BIOS) -> boot loader (e.g. GRUB) ->
kernel loads and initializes -> kernel starts PID 1 (init/systemd)
-> PID 1 brings up services -> login prompt.
2. PID 1 is the first process the kernel starts; it becomes the
ultimate parent because any process whose parent dies gets
"re-parented" to PID 1, which is responsible for reaping them so
they don't become permanent zombies.
3. It defends against a compromised or malicious boot loader/kernel
being loaded -- each stage cryptographically verifies the next
before running it, forming an unbroken chain of trust from firmware
to OS.
CHAPTER 22
1. A unit file describes how systemd should manage something (a
service, mount, timer, etc.); `systemctl start` runs it now,
`systemctl enable` additionally creates the symlink that makes it
start automatically at future boots.
2. A kernel module is code that can be loaded into the running kernel
without a reboot; dynamic loading avoids needing every possible
driver compiled into every kernel, and lets hardware support be
added or removed as needed.
3. Advantage: parallelized, dependency-aware startup made boot much
faster than SysVinit's sequential scripts. Criticism: it's a large,
complex system taking on much more than "just init," violating the
Unix "do one thing" philosophy in some critics' view.
CHAPTER 23
1. Most kernel-exposed resources -- regular files, directories, devices,
pipes, sockets, and process information (/proc) -- are all accessed
through the same read/write/open file-descriptor interface.
2. stdin (0), stdout (1), stderr (2).
3. Because /proc exposes process information as regular files/
directories (e.g. /proc/<pid>/status) -- ps just reads and parses
those files, with no special kernel API required beyond the normal
filesystem calls.
CHAPTER 24
1. /etc holds static configuration files; /var holds data that changes
at runtime (logs, databases, spool queues, caches).
2. /root is the superuser's home directory, kept separate from /home so
it remains accessible even if /home is a separate, unmounted, or
network filesystem during early boot/recovery.
3. The UsrMerge moved /bin, /sbin, /lib (and variants) to be symlinks
into their /usr equivalents, unifying what had become a historically
arbitrary split between "essential early-boot" and "everything else"
binaries, simplifying packaging and read-only-root setups.
CHAPTER 25
1. `-rwx r-x r--`: file type (-, regular file), owner has read/write/
execute, group has read/execute, others have read-only.
2. A hard link is a second directory entry pointing at the SAME inode
(indistinguishable from the "original," can't cross filesystems); a
symbolic link is a separate small file containing a path string,
which can cross filesystems and can point at nonexistent targets.
3. SUID makes a program run with its owner's (often root's) privileges
regardless of who invokes it. Legitimate use: `passwd`, so a normal
user can update the root-owned password file in a controlled way.
Risky use: a SUID-root script or binary with a shell-injection or
path bug becomes an instant privilege escalation.
CHAPTER 26
1. `cp` needs `-r` because copying a directory means recursively
copying its contents; `rm` deletes a bare file with no recursion
needed, but by convention still requires `-r` for directories --
the asymmetry the question points at is that `rm -rf` easily deletes
a whole tree with one flag combination, unlike cp's need for content
duplication logic.
2. A script computed a path using an unset/emptied environment
variable, which collapsed the intended path down to `rm -rf /`
equivalent, deleting content from the root of the filesystem.
3. Using `set -u` (fail on unset variables) and, for any risky rm/mv/
dd command, first replacing the destructive command with `echo` to
print exactly what WOULD run -- catching a wrong or empty variable
before it does damage.
CHAPTER 27
1. It inverts the match, printing lines that do NOT match the pattern
-- useful for filtering out noise (e.g. `grep -v DEBUG` on a log).
2. `sed 's/a/b/'` without the trailing `/g` flag only replaces the
FIRST match per line; adding `g` (global) replaces every match on
each line.
3. It's a real, documented exchange between two giants of computer
science (Doug McIlroy and Donald Knuth) where a short pipeline of
small Unix tools solved a text-processing task as effectively as a
much longer bespoke program -- a concrete demonstration of "small
tools, composed" over "one big program."
CHAPTER 28
1. It redirects file descriptor 2 (stderr) to wherever file descriptor
1 (stdout) currently points.
2. Order matters because redirections are applied left to right: `>
file 2>&1` first points stdout at the file, then points stderr at
wherever stdout NOW points (the file) -- both go to the file.
`2>&1 > file` first points stderr at wherever stdout CURRENTLY
points (the terminal), then redirects stdout to the file -- stderr
stays on the terminal.
3. A script fetched via `curl | sh` can change between the moment you
inspect it and the moment it executes (or serve different content
based on request headers/IP), so what you reviewed isn't guaranteed
to be what runs; the safer alternative is downloading it to a file,
reading it, then executing that file.
CHAPTER 29
1. `find` searches the live filesystem in real time (always accurate,
slower on large trees); `locate` searches a prebuilt index (very
fast, but can be stale if the index hasn't been updated since a
recent change).
2. `which` shows the path of the executable that would run based on
PATH lookup; `type` also reports if the command is actually a shell
builtin, alias, or function rather than an external binary.
3. Because filtering by modification/access time (`find / -mtime -1`)
quickly surfaces exactly which files changed around the time of a
suspected compromise or incident, narrowing an otherwise huge
filesystem down to a short, relevant list.
CHAPTER 30
1. Dependency metadata (what else it needs), version info, install
scripts (pre/post-install hooks), and a manifest of exactly which
files it installs (enabling clean removal).
2. Without automated dependency resolution, installing one piece of
software could require manually finding, building, and installing
an unpredictable chain of other libraries at compatible versions --
often conflicting with what other installed software needed.
3. `apt remove` uninstalls the package's files but leaves its
configuration files in place; `apt purge` also removes those
configuration files.
CHAPTER 31
1. `ps` gives a single point-in-time snapshot; `top` continuously
refreshes, showing live, changing resource usage.
2. Because %CPU is measured per core available -- a process using two
full cores on a multi-core machine shows 200%, not capped at 100%.
3. A fork bomb is a process that recursively spawns copies of itself
until it exhausts the system's process table/resources; because it
happens WHILE it's running, you need process tools (and often `kill`
with the right scope, or a ulimit on max processes) that still work
even as the system is being overwhelmed.
CHAPTER 32
1. SIGHUP (hang-up) -- historically sent when the controlling terminal
closes, and by default it terminates the process unless handled or
disowned.
2. It makes a process ignore SIGHUP and redirects its output to a file
(nohup.out by default), so it survives the terminal closing.
3. tmux/screen keep a persistent session you can detach from and
reattach to later (including to see live output and interact again),
whereas nohup only lets the process survive -- you can't get an
interactive session back.
CHAPTER 33
1. `Restart=always` tells systemd to automatically restart the service
if it exits for any reason; leaving it unset means a crash leaves
the service down until someone notices and restarts it manually.
2. `systemctl start` runs it now, one time; `systemctl enable` makes it
start automatically on future boots (they're independent -- you can
do either without the other).
3. Because `Restart=` wasn't configured, so after the process crashed,
nothing brought it back up, and no alert fired until a person or
monitor eventually noticed it was down.
CHAPTER 34
1. The journal is a structured, binary, indexed, queryable log store
managed by systemd (`journalctl`); traditional `/var/log` files are
plain text files each application/daemon writes to independently.
2. logrotate periodically compresses, renames, and eventually deletes
old log files based on size/age rules, so logs don't grow
unbounded and fill the disk.
3. Because a full disk isn't just "no more logs" -- once free space
hits zero, writes fail system-wide, which can crash databases,
block new connections, and break unrelated services that also need
to write to disk, turning a logging issue into a much bigger outage.
CHAPTER 35
1. sudo logs who ran what and when, and can be scoped to specific
commands per user, whereas a shared root login gives everyone
unrestricted, unattributable access.
2. A critical sudo heap-overflow vulnerability (found in 2021,
present for roughly a decade) that let a local unprivileged user
gain root; significant because sudo is nearly universally installed
and the flaw required no special configuration to exploit.
3. Because `visudo` validates the syntax before saving and locks the
file during editing -- a syntax error written directly to
/etc/sudoers can break sudo entirely, potentially locking everyone,
including yourself, out of privileged access.
CHAPTER 36
1. Capabilities let a process be granted ONE specific privileged
operation (e.g. binding to a low port, raw sockets) without needing
full root -- SUID-root grants ALL of root's power just to get one
privileged operation.
2. CAP_NET_RAW grants the ability to create raw sockets; ping needs it
(via SUID or the capability directly) to construct raw ICMP packets,
which is otherwise a privileged operation.
3. An ACL can grant specific permissions to specific additional users
or groups beyond the single owner and single group that basic Unix
permissions allow.
CHAPTER 37
1. Which network interface/gateway to send packets to, based on
destination IP/subnet.
2. Inside a container, loopback (127.0.0.1) is private to that
container's own network namespace -- it does NOT reach services in
sibling containers or the host, unlike on a normal host where
loopback reaches everything bound to it.
3. Check locally first (is the process even listening, `ss -tlnp`),
then local network path (routing, firewall), then DNS resolution,
then the remote host/service itself.
CHAPTER 38
1. `ss` replaced `netstat`, because it reads directly from kernel data
structures and is significantly faster on systems with many open
sockets.
2. `dig +short` gives just the resolved IP(s)/record value with no
extra formatting; the fuller output additionally shows query
timing, the authoritative server used, TTLs, and flags -- useful
when diagnosing DNS-specific issues rather than just "what does this
resolve to."
3. Check locally (is anything listening, `ss`), then routing/firewall
locally, then DNS resolution (`dig`), then reachability to the
remote host/port directly (`curl -v`, or `tcpdump` to see if packets
even arrive).
CHAPTER 39
1. `set -e` makes the script exit immediately if any command fails;
without it, a script continues past a failed command as if nothing
went wrong, potentially compounding the error.
2. `set -u` treats referencing an unset variable as an error and exits
immediately, which would have stopped a script from silently
collapsing a path down to `/` the way Chapter 26's incident did.
3. A deployment script accidentally activated old, dead feature-flagged
code on production servers because of a manual, error-prone,
partially-completed deployment process across many servers, not
simply "a bug in the trading logic" -- the failure was fundamentally
a deployment-process failure.
CHAPTER 40
1. Cron jobs run with a minimal environment (no login shell, a bare-
bones PATH, no interactive profile variables) -- a script relying on
PATH entries or environment variables set in your interactive shell
silently fails or behaves differently under cron.
2. Use absolute paths for every command and file, and explicitly set
any required environment variables (or source a known environment
file) at the top of the script rather than relying on inherited
shell state.
3. A dead man's switch expects a periodic "I'm alive and I ran
successfully" signal (e.g. a ping to a monitoring endpoint) and
alerts when that signal STOPS arriving -- it catches a cron job that
silently stops running entirely, which "alert on error output"
can't, since a job that never runs produces no error output either.
CHAPTER 41
1. It counts processes that are running OR waiting for CPU, and on
Linux also processes in uninterruptible sleep (waiting on I/O) --
so a high load average can reflect I/O pressure, not just CPU
demand.
2. I/O pressure -- processes are blocked waiting on disk/network I/O
rather than waiting for CPU time.
3. Because "load average: 8" means something very different on a
4-core machine (heavily oversubscribed) versus a 32-core machine
(mostly idle) -- it must be compared against core count to be
meaningful.
CHAPTER 42
1. `strace` shows the syscalls a process makes (and their arguments/
return values/timing); `lsof` shows what files, sockets, and other
descriptors a process currently has open -- one shows activity over
time, the other shows current state.
2. Check what it's doing right now (`strace -p`), what it has open
(`lsof -p`), and its process state (`ps`, is it D-state?) -- in
that rough order.
3. `df` reports free space based on the filesystem's own accounting;
`du` reports space actually used by files it can see and walk. They
disagree when a file is deleted but still held open by a running
process -- `lsof | grep deleted` resolves the mystery by finding
that file.
CHAPTER 43
1. `ulimit` sets per-process resource limits, configured per shell/
process and inherited by children; a cgroup limits and accounts
for an entire GROUP of processes together, enforced by the kernel
independent of any one process's own settings.
2. Because each incoming connection typically consumes at least one
file descriptor (a socket), and the default per-process open-file
ulimit (often 1024) is far lower than the number of concurrent
connections a busy server needs to hold open.
3. A container's memory/CPU limit (`docker run --memory`, or a
Kubernetes Pod's `resources.limits`) is implemented using exactly
this chapter's cgroup mechanism -- the container runtime creates a
cgroup and applies the limit to it.
CHAPTER 44
1. PID namespace, network namespace, mount namespace (with a root
filesystem), and a cgroup.
2. Because a container's processes are still ordinary Linux processes
running under the SAME host kernel, just with a namespaced view --
the host's process table sees all of them, just as it would see any
other process.
3. Docker did not invent namespaces or cgroups (both existed in the
Linux kernel beforehand, used by tools like LXC); Docker's real
contribution was packaging these primitives with an easy build/
ship/run workflow (Dockerfile, image format, registry) that made
containers accessible to ordinary developers.
CHAPTER 45
1. A VM virtualizes hardware -- each VM runs its own full kernel on
emulated/virtualized hardware; a container virtualizes the OS's
process view -- containers share one host kernel, isolated via
namespaces and cgroups.
2. Because starting a container just means starting a new process
(with namespaces applied) under an already-running kernel, whereas
a VM must boot an entire separate kernel and OS from scratch.
3. They trade some of a container's speed/density advantage for
stronger isolation (often via a VM-like boundary or intercepted
syscalls) -- a team chooses this when running less-trusted code
(e.g. multi-tenant workloads) where plain container isolation feels
insufficient.
CHAPTER 46
1. A layer is one immutable filesystem diff in an image; layers are
cacheable/shareable because multiple images built from a common base
or common early steps can reuse the exact same layer instead of
storing/downloading it again.
2. The writable layer holds all changes made while the container ran;
it is deleted along with the container when the container is
removed, unless that data lives in a volume.
3. Because Docker caches layers and invalidates a layer (and every
layer after it) as soon as its inputs change -- putting rarely-
changing dependency installation first means routine application
code changes don't force a slow dependency reinstall on every build.
CHAPTER 47
1. Checks locally for the image; if missing, pulls it (all its layers)
from the registry; creates a new container (namespaces, cgroup,
writable layer) from the image; starts the container's main
process.
2. `docker stop` sends SIGTERM (then SIGKILL after a timeout) to halt
a running container, but the container and its filesystem still
exist; `docker rm` deletes a (stopped) container and its writable
layer entirely.
3. Because `docker exec` starts an ADDITIONAL process inside the
container's namespaces -- the container's original main process
(PID 1 inside the container) is unaffected by that shell exiting.
CHAPTER 48
1. Because Docker caches each instruction's layer and invalidates
everything after the first changed layer -- installing dependencies
before copying frequently-changing application code means routine
code changes don't force a slow dependency reinstall.
2. It limits the blast radius of a container escape or a vulnerability
in the containerized application -- an attacker who breaks out as a
non-root user lands with far fewer host privileges than one who
breaks out as root.
3. It tells Docker which files/directories to exclude from the build
context sent to the daemon -- preventing large or sensitive files
(like `.git`, `node_modules`, or credentials) from being
unnecessarily included or accidentally baked into an image layer.
CHAPTER 49
1. It's deleted along with the container's writable layer -- gone
permanently unless it was stored in a volume or bind mount instead.
2. A named volume is managed by Docker itself (Docker decides where it
lives on the host, referenced by name); a bind mount points directly
at a path you specify on the host filesystem.
3. Because volumes are meant to potentially outlive any one container
(e.g. a database's data surviving a container recreate/upgrade) --
automatically deleting them with their last container risks silent,
irreversible data loss.
CHAPTER 50
1. It publishes the container's port 80 to the host's port 8080,
making it reachable from outside the host; without it, the port is
only reachable from other containers on the same Docker network (or
not at all, depending on network mode).
2. Docker runs an embedded DNS server for user-defined networks that
automatically resolves container names to their current IP,
updating as containers are created/removed.
3. The default bridge network is a legacy setup that predates this
embedded DNS feature and doesn't provide it -- user-defined networks
were introduced specifically to add automatic name resolution and
better isolation.
CHAPTER 51
1. It guarantees the dependency's CONTAINER has started, not that the
service inside it is actually ready to accept connections/requests.
2. Named volumes are preserved after `docker compose down`; only
containers and the default network are removed. `docker compose
down -v` additionally removes volumes.
3. Building a real readiness/health check into the dependent service's
startup logic (retry with backoff until the dependency responds) or
using Compose's `healthcheck` + `condition: service_healthy` --
this directly foreshadows Kubernetes readiness probes (Ch60).
CHAPTER 52
1. A tag (like `:latest` or `:v2`) can be reassigned to point at a
different image later; a digest is a cryptographic hash of the
image's exact content, so pinning by digest guarantees you always
get the exact same bytes.
2. Registering a package/image name deliberately similar to a popular,
trusted one (a typo or lookalike), hoping developers will
accidentally pull the malicious one instead -- it works because
people often type or copy image names without double-checking them
carefully.
3. It indicates Docker (or a verified organization) has reviewed and
vouches for that image's publisher/maintenance -- it is NOT a
guarantee the image is free of vulnerabilities, which still requires
active scanning.
CHAPTER 53
1. Scheduling containers across many machines, restarting failed
containers automatically, load-balancing traffic across replicas,
rolling out updates without downtime, and scaling based on demand --
none of which plain Docker on one machine does for you.
2. Kubernetes controllers continuously compare the cluster's actual
state to the desired state you declared, and take action to close
any gap -- this loop is how self-healing, scaling, and rollouts all
work, driven by the same underlying mechanism.
3. Google's internal cluster management system, Borg, directly inspired
Kubernetes' design (several of Kubernetes' original creators had
worked on Borg at Google).
CHAPTER 54
1. The control plane decides what SHOULD be running cluster-wide (API
server, etcd, scheduler, controller manager); a node is a machine
that actually RUNS the workloads (kubelet, container runtime,
kube-proxy).
2. The kubelet runs on each node and keeps existing Pods running
locally even if it temporarily can't reach the control plane --
already-scheduled workloads keep running through a control-plane
outage, just without new scheduling/scaling decisions.
3. Because already-running Pods, managed by the kubelet on each node,
keep serving traffic independently of the control plane -- a
control-plane issue mainly affects the ability to make NEW changes
(deploys, scaling, rescheduling), not already-running workloads.
CHAPTER 55
1. A Pod represents one or more containers that are always scheduled
and run together, sharing a network namespace; it's usually not
created directly because Pods aren't self-healing on their own --
if one dies, nothing recreates it unless a higher-level controller
manages it.
2. A Deployment manages ReplicaSets (creating a new one on each update);
a ReplicaSet ensures a specified number of identical Pod replicas
exist at all times.
3. Because a rolling update creates a NEW ReplicaSet for the new
version while scaling the OLD ReplicaSet down to zero rather than
deleting it immediately -- the old ReplicaSet is kept around
(scaled to 0) to support fast rollbacks.
CHAPTER 56
1. It gives a stable, unchanging network identity (an IP and DNS name)
in front of a set of Pods whose individual IPs change constantly as
they're created and destroyed.
2. It uses a label selector to continuously match Pods with matching
labels, and yes, this updates automatically as matching Pods come
and go -- no manual re-registration is needed.
3. ClusterIP is reachable only inside the cluster; NodePort additionally
exposes a port on every node's own IP; LoadBalancer additionally
provisions an external cloud load balancer routing to the service.
CHAPTER 57
1. Practically, they're used for different intent (config vs.
sensitive data), Secrets get some extra protections (like not being
shown in plain `kubectl get` output by default, and support for
encryption-at-rest when the cluster is configured for it), but
neither is truly encrypted by default -- both are equally readable
to anyone with API access.
2. Because base64 encoding LOOKS like security (the values aren't
plainly readable in raw manifest text) but is trivially reversible
by anyone with basic tooling -- leading many people to falsely
assume Secrets are automatically safe from anyone with cluster
access.
3. Enabling encryption-at-rest for etcd, restricting RBAC access to
Secrets, and/or using a dedicated external secrets manager (like
Vault or a cloud KMS-backed secrets integration) rather than relying
on Kubernetes Secrets alone.
CHAPTER 58
1. A request is what the scheduler guarantees is reserved for a Pod
when placing it on a node; a limit is the hard ceiling the Pod is
not allowed to exceed (enforced by the kernel's cgroup mechanism).
2. It's not an error -- it means no node currently has enough
unreserved capacity to satisfy the Pod's requests; it's the
scheduler correctly refusing to overcommit a node beyond what it
promised other Pods.
3. HPA changes how many Pod REPLICAS exist, based on metrics; Cluster
Autoscaler changes how many NODES exist, based on whether Pods are
unschedulable due to lack of capacity. You typically need both
together because HPA can create more Pods than existing nodes have
room for, and only Cluster Autoscaler can add the capacity to
actually run them.
CHAPTER 59
1. Lower the Pod's CPU request (may mean it gets throttled sooner) or
add more/bigger nodes (costs more) -- the trade-off is resource
sufficiency for THIS Pod versus cost/capacity for the cluster.
2. A container can crash before its logging library finishes
initializing, leaving current logs empty; `kubectl logs --previous`
shows the logs from the last (crashed) instance instead of the
current, freshly-restarted one.
3. The Service's label selector doesn't actually match the Pod's
labels, or the Pod's readiness probe is failing intermittently so
it's cycling in and out of the Service's endpoint list.
CHAPTER 60
1. A failed liveness probe causes Kubernetes to RESTART the container;
a failed readiness probe removes the Pod from the Service's traffic
endpoints WITHOUT restarting it.
2. It enables an outage where the app is actually broken (can't serve
real requests) but Kubernetes has no way to know, because both
liveness and (if it exists) readiness are checking something too
shallow (just "is the HTTP server alive") to reveal the real
problem.
3. Because the rollout only proceeds to remove more old Pods once new
Pods pass their readiness probe -- if new Pods never become ready,
Kubernetes correctly stops there rather than continuing to replace
working old Pods with broken new ones.
CHAPTER 61
1. It re-applies the exact rendered Kubernetes YAML that was generated
and applied during revision 3's `helm install`/`upgrade` -- Helm
stores each revision's rendered manifest.
2. Because the next `helm upgrade` re-applies Helm's own tracked state,
silently overwriting the manual `kubectl edit` change with no
warning that it's discarding something.
3. Because it shows exactly what Kubernetes objects, RBAC permissions,
and resource settings the chart would create, letting you review
for problems (missing limits, excessive permissions, disabled
persistence) before anything is actually applied to your cluster.
CHAPTER 62
1. It tells you the program touched many pages that were already in
RAM but not yet mapped into its own address space (common and
cheap, e.g. shared libraries, copy-on-write pages) -- generally not
a concern on its own, unlike major faults which indicate disk I/O.
2. Because each syscall has fixed kernel-transition overhead regardless
of how much data it moves -- doing the same total I/O in fewer,
larger reads amortizes that fixed overhead across far more bytes
per call.
3. The program may load the file plus significant additional
structures (e.g. a hash map of every word) into memory
simultaneously, or it may buffer/duplicate data during processing --
either can push total memory usage well past the raw file size.
CHAPTER 63
1. Because the final stage only copies the already-built artifact from
the builder stage, discarding the builder stage's compilers, build
tools, and intermediate files entirely -- those never become part
of the final image's layers.
2. Because `depends_on` only waits for the dependency's CONTAINER to
start, not for the database process inside it to finish its own
startup and begin accepting connections; fix with an application-
level retry/backoff on connect, or a Compose healthcheck combined
with `condition: service_healthy`.
3. The base image's default user might already be root and the
Dockerfile's `USER` instruction may be placed before another
instruction (or a later stage in a multi-stage build) that resets
it, or `docker exec` itself may have been run with an explicit
`--user root` override.
CHAPTER 64
1. Because it's the only way to genuinely verify the rollout safety
mechanism (readiness gating, stalled rollout, rollback) actually
works, rather than assuming it does -- deploying only working
versions never exercises the failure path at all.
2. A readiness probe failure keeps old, working Pods serving traffic
while new ones are blocked -- users see no impact; a liveness probe
failure restarts containers, which, if misconfigured, can cause
repeated restarts and real user-visible disruption.
3. It assumes the Pod needs a very small, effectively negligible amount
of each resource, which can cause the scheduler to over-pack a node,
leading to real resource contention once the Pod's actual usage
turns out to be much higher.
CHAPTER 65
1. Because restoring service quickly limits user impact, and a known-
good previous version is a fast, low-risk action -- continuing to
investigate live while users are affected extends the outage for
the sake of understanding that can just as well happen after
service is restored.
2. It focuses on what the SYSTEM and PROCESS allowed to happen (missing
alerts, unclear runbooks, a gap in testing) rather than which
individual made a mistake -- assuming good faith and treating the
incident as a systems-design learning opportunity.
3. Config/selector/probe-path/scheduling-related failures (a, c, d, e)
would generally show up quickly in `describe pod`'s Events; a slow
memory leak leading to eventual OOMKilled (b) or a disk-fill (f)
might require watching trends over time or checking application
logs/metrics before the root cause is obvious.
CHAPTER 66
1. Configuration drift is the gap between the intended baseline and the
actual state of one or more machines: package versions, sysctls,
mounts, unit files, kernel versions, runtime state, or manual fixes.
2. A known-good host narrows the search space. Differences between good
and bad hosts often reveal partial rollouts, missing packages,
different kernels, changed mount options, or runtime drift faster
than reading every log line on the bad host.
3. A mitigation reduces immediate impact; a root cause explains why the
failure happened. Restarting a service may mitigate an outage, but it
does not explain the leak, dependency failure, or bad rollout that
made the restart necessary.
CHAPTER 67
1. systemd starts a service with a different environment, working
directory, user, PATH, limits, dependencies, and sandboxing than your
interactive shell.
2. `systemctl cat` shows the effective unit file plus drop-in overrides;
`status` shows current state, recent logs, and process information.
3. Hardening settings change what the process can read, write, execute,
and access. They can prevent real attacks, but they can also break
legitimate behaviour if untested.
CHAPTER 68
1. A backup job only proves bytes were written somewhere. A tested
restore proves you can recover the system or data when it matters.
2. You need the device, partition/LVM/RAID/encryption stack, filesystem
type, mount point, current usage, backup state, and whether the
filesystem supports online growth.
3. If a process still has the deleted file open, the directory entry is
gone but the disk blocks remain allocated until that file descriptor
is closed.
CHAPTER 69
1. The service may be bound only to loopback (`127.0.0.1`), blocked by a
firewall/security group, routed incorrectly, or listening in a
different network namespace.
2. A network namespace isolates interfaces, addresses, routes, firewall
state, and sockets, giving a process its own view of the network.
3. Packet capture is powerful but can be noisy and intrusive. Often
`ss`, routing, DNS, and firewall inspection identify the layer before
you need packets.
CHAPTER 70
1. The symptom tells you which layer to measure. Without it, you can
collect impressive data that does not answer the user-impacting
question.
2. A flame graph shows where time is spent across call stacks, making
hot paths visible without reading the whole codebase first.
3. Host metrics describe machine/resource behaviour; distributed traces
show how one request moves through services and where time/errors
occur along that path.
CHAPTER 71
1. Capabilities allow a narrow privilege, such as binding a low port,
without granting every privilege that comes with full root.
2. Mode bits mostly answer owner/group/other access. SELinux/AppArmor
can enforce policy based on process domain/profile, file labels,
paths, capabilities, and allowed behaviours even when Unix mode bits
would otherwise permit access.
3. Logs are useful only if they answer a detection or investigation
question and someone or something reviews them. Unread logs are
storage, not control.
CHAPTER 72
1. Restarting can erase volatile evidence: process state, memory growth,
open files, socket state, stack traces, and the exact failure mode.
2. Mitigation reduces immediate impact; remediation fixes the specific
defect; prevention changes tests, alerts, architecture, capacity, or
process so the same class of failure is less likely or easier to
catch.
3. Add the commands that worked, healthy/bad examples, known safe
actions, escalation criteria, rollback steps, and any alert or
dashboard links that would have shortened the incident.
CHAPTER 73
1. G = goroutine (your code), M = an OS thread, P = a processor slot
allowed to run Go code; there are GOMAXPROCS Ps, any number of Gs,
and as many Ms as currently needed.
2. A goroutine waiting on a socket is parked and its descriptor is
registered with epoll (the netpoller); one thread waits for all of
them and wakes only the goroutines whose sockets become ready.
3. Regular-file I/O can't use epoll on Linux, so a read on a stalled
disk or NFS mount blocks its OS thread inside the kernel; the
runtime starts more threads to keep running other goroutines.
CHAPTER 74
1. The kernel re-parents orphaned processes to PID 1 (of their PID
namespace), so PID 1 is the only process that can wait() for them.
2. It runs a tiny init (tini) as PID 1, which reaps zombies and forwards
signals to your program, which is no longer PID 1.
3. So the child and everything it started share a process group, and a
kill to the negative PGID reaches grandchildren too.
CHAPTER 75
1. Quota and period: microseconds of CPU time allowed per period, e.g.
150000 100000 = 150 ms per 100 ms = 1.5 CPUs.
2. Throttling is decided per 100 ms period: many threads can use the
whole quota early in a period and then stall for the rest, even
though the average over a second is below the limit.
3. Go 1.25+ reads the cgroup CPU limit and sets GOMAXPROCS to it, rounded
up, minimum 2, re-checked periodically; the GOMAXPROCS environment
variable overrides it.
CHAPTER 76
1. At roughly twice the live heap: about 400 MB.
2. It is a target the GC works towards (by collecting more often), not
a cap the runtime enforces; live data larger than the limit still
grows past it.
3. 137 = 128 + 9: the process was killed by SIGKILL, typically by the
kernel's OOM killer when the cgroup reached memory.max.
CHAPTER 77
1. Only that the bytes were copied into the kernel's page cache; they
may not be on stable storage yet.
2. rename(2) is atomic only within one filesystem; across filesystems
it becomes a copy that can be interrupted.
3. Losing the rename itself in a crash -- the directory entry change
is metadata that is only durable once the directory is fsynced.
CHAPTER 78
1. The soft limit is enforced; the hard limit is the ceiling an
unprivileged process may raise its own soft limit to.
2. It raises its soft RLIMIT_NOFILE to the hard limit minus one, and
restores the original soft limit for processes it starts.
3. All network descriptors are registered with one epoll instance;
waiting goroutines are parked, and the runtime is woken only for
descriptors that are ready.
CHAPTER 79
1. The command name (field 2) is in parentheses and may contain spaces
and parentheses, so only the LAST ')' reliably marks its end.
2. Read utime+stime (fields 14 and 15) twice, subtract, divide by
USER_HZ (100) and by the interval in seconds, times 100.
3. Kernel threads have no user-space command line; tools show their
name from the comm field in square brackets instead.
CHAPTER 80
1. CLONE_NEWPID (the child is PID 1 of a new PID namespace); CLONE_NEWUTS
(its own hostname).
2. /proc reflects the PID namespace it was mounted in; without a fresh
mount, ps inside would show the host's processes.
3. The cgroup's pids.max = 20; the refusals are counted in pids.events
("max N") in the cgroup directory.
CHAPTER 81
1. flat = time spent in the function itself; cum = time in the function
plus everything it called.
2. The goroutine profile; "489 @ ..." means 489 goroutines share exactly
that stack -- here, all blocked at the same line.
3. Time inside the kernel (TCP stack, page faults, scheduler) and in
non-Go code, using hardware performance counters.
CHAPTER 82
1. GOMEMLIMIT (Chapter 76), relying on the CPU limit for GOMAXPROCS
(Chapter 75), a SIGTERM-aware drain with preStop and a grace period
(Chapter 74), CGO_ENABLED=0 for a static distroless binary (Chapter 48).
2. Distroless images contain no /bin/sleep and the server has no sleep
flag, so an exec hook would fail; the built-in sleep action needs
nothing in the image.
3. With DynamicUser the UID changes on every start, so no file owner can
be set in advance; LoadCredential copies the root-only files into a
per-service directory only this service can read.
CHAPTER 83
(Personal/reflective -- no single correct answer; the value is in
articulating your own reasoning using this guide's vocabulary.)
3. A service mesh typically adds mutual TLS between services,
fine-grained traffic routing/splitting (e.g. canary percentages),
and detailed per-request observability -- none of which a plain
Service's simple label-based load balancing provides on its own.
CHAPTER 84
1. Because process/memory/syscall fundamentals have been stable for
decades and change slowly, while Kubernetes, container runtimes, and
distributions regularly change APIs, defaults, and operational
mechanics.
2. E.g. the Kubernetes deprecated API migration guide, checked before
an upgrade -- avoids a manifest using a removed `apiVersion` silently
failing to apply after the upgrade.
3. The underlying cgroup mechanism switched from cgroups v1's separate
per-resource hierarchies to v2's single unified hierarchy; some
memory-accounting edge cases changed as a result, so containers
could hit OOM-killed at effectively different real memory thresholds
than before, purely from the OS-level cgroup version underneath
them changing, without any change to the container's own configured
limit.