Chapter 67 -- systemd beyond start/stop: dependencies, timers, sandboxing, and debugging

In one sentence

systemd is not only a service launcher; it is a dependency engine, logger, supervisor, timer system, resource-control interface, and practical hardening tool.

Why this matters

Many production failures are "the process is fine, but the unit is wrong": wrong working directory, missing environment, bad restart policy, dependency cycle, timeout too short, service starts before the network/storage is truly ready, or a hardening option blocks a path the app needs.

How it actually works

Useful commands for real diagnosis:

systemctl status myapp.service
journalctl -u myapp.service -b --no-pager
systemctl cat myapp.service
systemd-analyze verify ./myapp.service
systemd-analyze critical-chain myapp.service
systemctl show myapp.service -p Restart -p User -p Group -p ExecStart

Useful unit-file ideas:

[Unit]
Description=Example API
After=network-online.target
Wants=network-online.target

[Service]
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server
Restart=on-failure
RestartSec=5s
TimeoutStartSec=30s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp /var/log/myapp

[Install]
WantedBy=multi-user.target

Do not cargo-cult hardening directives. ProtectSystem=strict is excellent when the write paths are explicit; it is an outage when the service legitimately needs to write somewhere you forgot to list.

Real-world scenario

A service starts manually but fails under systemd. The usual cause is not "systemd is broken"; it is that the interactive shell had environment variables, current directory, PATH, open files, or permissions the unit did not have.

Try it yourself

Create a disposable service that prints its environment:

systemd-run --user --unit env-demo /usr/bin/env
journalctl --user -u env-demo --no-pager
systemctl --user status env-demo

Compare that environment with your interactive shell:

env | sort | head

Common mistakes

Using Restart=always for a program that exits successfully by design.
Editing /lib/systemd/system/... instead of using an override under /etc/systemd/system/....
Forgetting systemctl daemon-reload after changing unit files.
Adding hardening flags without testing realistic application behaviour.

Check yourself

Why can a program work in your shell but fail as a systemd service?
What does systemctl cat show that systemctl status does not?
Why should hardening settings be tested like code?