Chapter 65 -- Project 4: diagnose and fix a planted incident, end to end

In one sentence

Have a friend (or your future self, a week later, having forgotten the details) deliberately break your Project 3 deployment in one specific way, without telling you what they changed, and practice the full incident process: detect, diagnose, mitigate, fix, and write a short postmortem.

Why this matters

Every skill in this guide -- OS internals, Linux commands, container and Kubernetes operation -- exists to be used under exactly this condition: something is broken, you don't know what, and people are waiting on you. This project is the only one in this guide that simulates that condition honestly, because in Projects 1-3 you always already knew what you had just changed. Practicing diagnosis on a problem someone else planted, with no hints, is the closest a self-study guide can get to a real on-call page -- and it is exactly what SRE and platform interviews increasingly test for directly (major companies run live "debug this broken cluster" exercises as part of their SRE interview loops).

How it actually works

   SETUP (ask a friend, a study partner, or do this to your own
   deployment a week later after you've forgotten the details):
   Introduce exactly ONE of these planted failures, chosen at random,
   into the Project 3 Kubernetes deployment:

     a) Change the readiness probe's path to one that 404s
     b) Set a memory limit lower than the app actually needs, causing
        OOMKilled restarts under load
     c) Change a ConfigMap value the app needs (e.g. a database host
        name) to something wrong
     d) Change the Service's label selector so it no longer matches
        the Deployment's Pods
     e) Set an overly aggressive livenessProbe (very short
        initialDelaySeconds) causing restart-loop on a normally-slow-
        starting app
     f) Fill the node's disk (or a PVC) to force a storage-related
        failure

   YOUR PROCESS, once told only "the app is down, users are getting
   errors":

     1. DETECT   -- confirm the actual user-visible symptom first
                    (curl the Service / Ingress yourself)
     2. DIAGNOSE -- run the Chapter 59 sequence: get pods, describe,
                    logs, events. Form a hypothesis before changing
                    anything.
     3. MITIGATE -- stop the bleeding first. Often this means
                    `kubectl rollout undo` rather than root-causing
                    live, if a previous known-good revision exists.
     4. FIX      -- once mitigated, identify and fix the actual root
                    cause (not just the symptom).
     5. WRITE    -- a half-page postmortem: what broke, how you
                    detected it, what you tried that didn't help,
                    what actually fixed it, and one concrete change
                    (a monitor, an alert, a stricter code review
                    check) that would have caught this earlier.

Real-world example

The five-step process above -- detect, diagnose, mitigate, fix, write -- is a direct, simplified mirror of the incident response process used at Google, Amazon, and most companies with a mature SRE practice, as documented in Google's Site Reliability Engineering book (the "mitigate first, root-cause second" ordering in particular is one of its most emphasized and most frequently learned-the-hard-way lessons: engineers under pressure to find the "real" cause often delay an available, working mitigation like a rollback, extending an outage's user impact for the sake of intellectual completeness that could have waited). Blameless postmortems -- the practice of writing up what happened without assigning personal fault, focusing instead on what the system and process allowed to happen -- is likewise a direct practice from that same book and from Etsy's widely cited "Blameless PostMortems and a Just Culture" (2012) engineering blog post, one of the most influential single blog posts in the SRE field's history. Practicing the format here, on a low-stakes toy incident, is what makes writing one under real pressure feel familiar instead of foreign.

Try it yourself

This chapter's project is the "try it yourself." If no study partner is available, write all six planted failures (a-f above) on slips of paper, draw one at random a week from now without looking, apply it to your Project 3 deployment, and force yourself to diagnose it cold.

Common mistakes

Skipping straight to "fix the root cause" under time pressure instead of mitigating first -- in a real incident, this is the single most common mistake that extends outage duration.
Guessing and changing multiple things simultaneously ("let me just restart everything and bump the memory limit and check the ConfigMap") -- this can accidentally fix the problem while leaving you unable to explain why, which defeats the exercise and, in a real incident, often introduces a second unrelated problem.
Writing a postmortem that focuses on blame ("I should have checked this sooner") rather than systemic prevention ("we should have an alert that fires when readiness-probe failures exceed N for M minutes").
Not timing yourself. Real incidents have a clock running; practicing without one removes the exact pressure this exercise exists to simulate.

Check yourself

Why does "mitigate first, root-cause second" often mean choosing kubectl rollout undo over continuing to investigate, even if you haven't yet found the actual root cause?
What makes a postmortem "blameless" in practice -- what does it focus on instead of individual fault?
Of the six planted-failure categories listed, which ones would the Chapter 59 describe pod Events block surface almost immediately, and which ones would require checking application logs or external symptoms first?