Chapter 34 — Logs: `journalctl` and where output really goes

In one sentence

On a systemd-based system, most service output is captured centrally by the journal, queryable with journalctl — but plenty of software still writes plain text log files under /var/log, and knowing which is which (and keeping either from filling your disk) is a real, constant operational skill.

Why this matters

A disk that fills up with unrotated logs is one of the single most common, most preventable real production outages in the industry's entire history — and it's caused by a genuinely simple oversight.

How it actually works

   journalctl                    # ALL journal entries, oldest first
   journalctl -u myapp            # just THIS service's log entries
   journalctl -f                   # FOLLOW -- live tail, like `tail -f`
   journalctl -b                    # just THIS boot session
   journalctl --since "1 hour ago"   # time-filtered
   journalctl -p err                # only ERROR-level and above

   # Traditional plain-text logs still live under /var/log -- not
   # everything goes through the journal (Chapter 24's FHS: /var is
   # "variable data," logs are the classic example)
   tail -f /var/log/syslog          # (Debian/Ubuntu) or
   tail -f /var/log/messages         # (RHEL/Fedora family) -- the
                                    # general system log
   ls /var/log/                      # see what else is being logged --
                                    # often includes per-application
                                    # logs, auth logs, and more

Log rotation (logrotate, the standard tool) exists specifically to prevent logs from growing forever:

   /etc/logrotate.conf and /etc/logrotate.d/*  define, per log file/
   category: how often to rotate (daily/weekly), how many old copies to
   KEEP, whether to compress old copies, and when to DELETE the oldest
   ones entirely. Without this running correctly, a busy service's logs
   genuinely can, and regularly do, grow without bound.

Real-world example

   "The disk filled up with log files and took down production" is one
   of the most common, most WIDELY DOCUMENTED real operational
   incidents across the entire software industry -- a search of almost
   any large company's public postmortem archive turns up at least one
   example, and it's a recurring theme in SRE training material
   precisely BECAUSE it keeps happening, in new forms, across different
   organisations, for decades. The shape is almost always the same: a
   service (often a NEW one, or one recently changed to log more
   verbosely for debugging) writes logs that were never added to
   `logrotate`'s configuration, or a misconfigured log rotation rule
   that silently stops working, or a runaway ERROR condition that
   causes MASSIVE log volume in a short period (an application stuck in
   a tight retry-and-log loop is a classic culprit) -- and because
   Linux behaves VERY badly when its root filesystem runs completely
   out of space (new file writes fail, and depending on what else lives
   on that filesystem, entirely unrelated services can start failing
   too), what began as "just a logging problem" becomes a full outage.
   THE PREVENTABLE FIX, universally recommended across incident
   postmortems on this exact topic: EVERY log-producing service gets an
   explicit logrotate configuration from day one (not added later, after
   the first incident), PLUS disk-usage monitoring with alerting well
   BEFORE the disk is actually full (Chapter 41 covers monitoring disk
   usage properly) -- treating log rotation as a deployment checklist
   item, not an afterthought.

Try it yourself

# Explore your own system's logs
journalctl -u ssh --no-pager 2>/dev/null | tail -10 || \
  journalctl -u sshd --no-pager 2>/dev/null | tail -10

journalctl --since "10 minutes ago" --no-pager | tail -20

ls -la /var/log/ | head -15
du -sh /var/log/* 2>/dev/null | sort -rh | head -10   # biggest log
                                                       # consumers right
                                                       # now

# See your log rotation configuration
cat /etc/logrotate.conf 2>/dev/null | head -20
ls /etc/logrotate.d/ 2>/dev/null

# See how much disk space is currently used, and by what
df -h /
du -sh /var/log 2>/dev/null

Common mistakes

Deploying a new, verbose service without checking it's covered by log rotation. The single most common preventable cause of "disk filled with logs" incidents.
Only monitoring "is the service up," not "is the disk filling up." A disk-full incident is often entirely predictable hours in advance if disk-usage trends are actually being watched.
Assuming journalctl captures EVERYTHING. Plenty of software still writes its own plain-text log files directly to /var/log, entirely outside the journal — check both when troubleshooting.

Check yourself

What's the difference between the systemd journal and traditional /var/log files?
What does logrotate actually do, and why does it exist?
Why does "the disk filled up with logs" so often cascade into a much bigger outage than just "logging stopped working"?

Chapter 34 — Logs: journalctl and where output really goes