Chapter 72 -- Incident response and refactoring on real Linux systems

In one sentence

The senior Linux workflow is: stabilize, preserve evidence, understand the failure, fix the smallest safe thing, verify, then refactor the system so the same class of failure is less likely or easier to diagnose next time.

Why this matters

Real incidents are not exam questions. You may need to restore service before you fully understand the root cause, but every emergency action should be visible, reversible when possible, and followed by cleanup.

How it actually works

Production-safe incident rhythm:

   1. State the user-visible impact.
   2. Freeze context: what changed, when, on which hosts?
   3. Capture volatile evidence before restarting: logs, process state,
      sockets, open files, resource usage.
   4. Mitigate using the smallest reversible action.
   5. Verify recovery from the user's perspective.
   6. Root-cause with evidence.
   7. Refactor: tests, alerts, runbooks, config, capacity, rollback path.

Evidence commands:

date -Is
uptime
free -h
df -hT
ss -s
systemctl --failed
journalctl -p warning -b --no-pager | tail -100

Refactoring examples:

   repeated disk-full incident
      -> add log rotation, quota/retention, filesystem alert, runbook

   repeated OOM kill
      -> measure working set, fix leak or limit, add memory dashboard,
         tune request/limit, add load test

   repeated manual service restart
      -> fix readiness check, dependency ordering, backoff, crash alert,
         and deploy rollback

Real-world scenario

An API recovers every time someone restarts it, so the team adds a cron restart. That hides the leak and destroys evidence. A better response is to capture RSS, heap/profile data if available, request rate, open file count, and OOM/restart events; then choose a temporary mitigation while preserving enough information to fix the leak.

Try it yourself

Write a one-page runbook for "service is slow" using Appendix C as a starting point. For each step, write:

   command to run
   what healthy looks like
   what bad looks like
   what action is safe
   what action needs approval

Common mistakes

Restarting before capturing the evidence that would explain the incident.
Treating a one-off manual fix as done without adding a test, alert, or configuration change.
Making several changes at once and losing the ability to know what helped.
Writing a postmortem that blames a person instead of improving the system.

Check yourself

Why can a restart make root-cause analysis harder?
What is the difference between mitigation, remediation, and prevention?
What should be added to a runbook after an incident?