Chapter 33 — systemd and services: how "always running" actually works

In one sentence

A systemd service is a small, declarative file describing how to start a program, when to start it, and (crucially) whether to restart it if it crashes — turning "keep this running forever" from a manual chore into a guarantee the init system enforces.

Why this matters

The difference between a service that silently stays down after a 3am crash and one that restarts itself automatically is, very often, a single line in a unit file that nobody set.

How it actually works

   # /etc/systemd/system/myapp.service -- a minimal, real unit file
   [Unit]
   Description=My Application
   After=network.target              # start AFTER networking is up

   [Service]
   ExecStart=/usr/bin/myapp --config /etc/myapp/config.yml
   Restart=always                    # THE critical line -- restart on
                                     # ANY exit, crash or otherwise
   RestartSec=5                       # wait 5 seconds before restarting
   User=myappuser                     # don't run as root unless you
                                     # genuinely need to

   [Install]
   WantedBy=multi-user.target         # start automatically at normal
                                     # boot

   sudo systemctl daemon-reload        # tell systemd to re-read unit
                                      # files after you edit one
   sudo systemctl enable myapp         # start automatically on boot
   sudo systemctl start myapp           # start it right now
   sudo systemctl status myapp          # is it running? recent log
                                      # lines (Chapter 34)?
   sudo systemctl restart myapp          # stop then start

The Restart= options that matter most:

   no          (the default if unset) never restart automatically
   always       restart no matter HOW it exited -- crash, clean exit,
               anything
   on-failure   restart only if it exited with a non-zero status/signal
               (Chapter 3) -- NOT if it exited cleanly (status 0)

Real-world example

   A well-documented, extremely common real production failure pattern,
   discussed across countless postmortems and SRE writeups: a service
   is deployed and manually started (`systemctl start myapp`) but never
   actually `enable`d, and its unit file never sets `Restart=`. It runs
   fine for weeks or months. Then, one night, the process crashes --
   an unhandled exception, an out-of-memory kill (Chapter 13), a
   transient bug -- and because there's no `Restart=` directive, systemd
   makes NO attempt to bring it back. The service simply stays down,
   silently, until a human notices (often hours later, via a customer
   complaint or a downstream alert, rather than the actual crash itself)
   and manually restarts it.
   THIS EXACT GAP -- a real, common finding in real production
   incident postmortems across the industry -- is why `Restart=always`
   (or, more carefully considered, `Restart=on-failure` if a clean exit
   genuinely should stay stopped) is treated as close to a DEFAULT
   BEST PRACTICE for any service expected to run continuously, and why
   `systemctl enable` (survive a REBOOT) and setting a restart policy
   (survive a CRASH) are both treated as basic, non-optional hygiene for
   anything running in production -- two separate, easy-to-forget
   settings, each addressing a different real failure mode.

Try it yourself

# Write and install a tiny real service (safe, in your lab)
cat > /tmp/heartbeat.sh << 'EOF'
#!/bin/bash
while true; do
  echo "heartbeat: $(date)"
  sleep 5
done
EOF
chmod +x /tmp/heartbeat.sh

sudo tee /etc/systemd/system/heartbeat.service > /dev/null << 'EOF'
[Unit]
Description=Heartbeat demo service
After=network.target

[Service]
ExecStart=/tmp/heartbeat.sh
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start heartbeat
sleep 6
sudo systemctl status heartbeat --no-pager   # see it running

# Prove Restart=always works: kill it, watch it come back
sudo kill $(pgrep -f heartbeat.sh)
sleep 4
sudo systemctl status heartbeat --no-pager   # should be running again,
                                             # with a new PID

# Clean up
sudo systemctl stop heartbeat
sudo systemctl disable heartbeat 2>/dev/null
sudo rm /etc/systemd/system/heartbeat.service /tmp/heartbeat.sh
sudo systemctl daemon-reload

Common mistakes

systemctl start without systemctl enable. Works fine until the next reboot, then the service simply doesn't come back — a very common real cause of "it worked yesterday, why is it down after maintenance."
No Restart= policy on a production service. Silent, extended downtime after any crash, with no automatic recovery.
Forgetting daemon-reload after editing a unit file. systemd caches unit file contents — your edits won't take effect until you tell it to re-read them.

Check yourself

What does Restart=always do, and what's the risk of leaving it unset?
What's the difference between systemctl start and systemctl enable?
Why did the service in the real-world example stay down for hours after crashing?