Part 12 — Shell scripting and automation

You've now run dozens of commands by hand across Parts 7–11. This part is about not doing that by hand anymore — enough bash to automate the repetitive parts of everything covered so far, from file operations to process management to basic networking checks.

Chapter 39 — Bash scripting fundamentals

In one sentence

A bash script is just a sequence of the same commands you'd type interactively, saved to a file and given a few extra safety rails — and those safety rails (set -e, quoting, checking exit status) are what separates a script you can trust in production from one that fails silently.

Why this matters

An unattended script that keeps running after something has already gone wrong is one of the most dangerous, most common real sources of production damage — automation multiplies whatever mistake you make, at machine speed.

How it actually works

   #!/bin/bash              # the SHEBANG -- tells the OS which
                            # interpreter to run this file with
                            # (Chapter 6's exec(), essentially)
   set -e                    # EXIT IMMEDIATELY if any command fails
                            # (non-zero exit status, Chapter 3) --
                            # without this, a script just KEEPS GOING
                            # after a failure, often compounding it
   set -u                    # error out on any UNSET variable, instead
                            # of silently treating it as empty (this is
                            # EXACTLY the class of bug behind Chapter
                            # 26's Steam `rm -rf` incident)
   set -o pipefail            # a pipeline's exit status reflects the
                            # FIRST failing command, not just the last
                            # one -- without this, `false | true` reports
                            # SUCCESS, hiding a real failure upstream

   # a common, recommended combination:
   set -euo pipefail

   VARIABLE="value"             # no spaces around =, by convention
   echo "$VARIABLE"              # ALWAYS quote variable expansions --
                                 # unquoted expansion is a classic
                                 # source of bugs with spaces/globs

   if [ "$1" = "start" ]; then    # test/condition syntax
     echo "starting"
   elif [ "$1" = "stop" ]; then
     echo "stopping"
   else
     echo "usage: $0 start|stop" >&2
     exit 1
   fi

   for file in *.txt; do          # loop over files
     echo "processing $file"
   done

   result=$(some_command)          # capture command output into a
                                  # variable
   if [ $? -ne 0 ]; then           # $? = the PREVIOUS command's exit
                                  # status (Chapter 3) -- but prefer
                                  # `set -e` over manually checking
                                  # this everywhere
     echo "failed" >&2
     exit 1
   fi

Real-world example

   Knight Capital Group's 2012 trading incident remains one of the
   most widely studied, most consequential real illustrations of why
   AUTOMATION AND DEPLOYMENT DISCIPLINE matters -- publicly documented
   via the SEC's own investigative order (a real, official regulatory
   document, not just press reporting). During a software deployment,
   NEW trading code was rolled out to SEVEN production servers, but --
   due to a deployment process failure -- ONE of the eight servers
   retained OLD, previously-repurposed code that reused a flag value
   with a different, dangerous meaning under the new system. When live
   trading began, that ONE server's stale code triggered a RUNAWAY
   sequence of unintended trades. Automated systems, running exactly as
   designed, executed the resulting bad instructions at high speed with
   NO human in the loop fast enough to intervene -- Knight Capital lost
   roughly $440 MILLION in about 45 minutes, and the firm itself was
   sold off shortly afterward as a direct result.
   THE LESSON THIS CHAPTER TAKES FROM IT, stated precisely: the failure
   wasn't one bad line of code in isolation -- it was a DEPLOYMENT
   PROCESS with no verification step confirming all servers actually
   ran the SAME, INTENDED code before going live, combined with
   automation that had no circuit breaker to catch and halt an
   obviously-anomalous pattern of behaviour once it started. `set -e`
   catching a script failure immediately, verifying deployment state
   before proceeding, and building in sanity checks BEFORE automation
   acts irreversibly are all, in spirit, the exact same discipline this
   incident demonstrates the cost of skipping -- at a scale ordinary
   engineers will rarely if ever match, but the underlying principle
   scales down to every cron job and deploy script you'll ever write.

Try it yourself

cd ~/lab/playground

# See set -e catch a failure that would otherwise continue silently
cat > unsafe.sh << 'EOF'
#!/bin/bash
echo "step 1"
false                      # this "fails" but the script keeps going
echo "step 2 (ran anyway, even though step 1 failed!)"
EOF
chmod +x unsafe.sh
./unsafe.sh

cat > safe.sh << 'EOF'
#!/bin/bash
set -euo pipefail
echo "step 1"
false                      # this time, the script STOPS here
echo "step 2 (this line should never print)"
EOF
chmod +x safe.sh
./safe.sh; echo "exit status: $?"

# See set -u catch an unset variable (the Steam-bug class of mistake)
cat > unset_demo.sh << 'EOF'
#!/bin/bash
set -u
echo "TARGET is: ${TARGET}"
EOF
chmod +x unset_demo.sh
./unset_demo.sh    # errors out immediately, loudly -- exactly what you
                   # want, instead of silently treating TARGET as empty

rm unsafe.sh safe.sh unset_demo.sh

Common mistakes

Writing a script without set -euo pipefail. The single highest- leverage habit in this whole chapter — add it to every script, as a default, before you write anything else.
Leaving variable expansions unquoted ($VAR instead of "$VAR"). A value containing spaces or shell-special characters can silently break the script's logic in ways that are hard to spot by eye.
Trusting a script "worked" because it didn't print an error. Without set -e, a failed command's error can scroll past unnoticed while later steps run anyway, on bad assumptions.

Check yourself

What does set -e do, and what's the risk of a script WITHOUT it?
Why does set -u specifically address the class of bug behind Chapter 26's Steam rm -rf incident?
What was the actual root cause of the Knight Capital incident — not just "a bug," but specifically?