Chapter 68 -- Storage operations: LVM, RAID, filesystem growth, backups, and recovery

In one sentence

Storage expertise is knowing the stack from block device to filesystem to application durability, and testing restore paths before an outage proves they do not work.

Why this matters

Disk incidents are unforgiving. A CPU spike usually gives you time to think; a full filesystem, failed disk, corrupt journal, or accidental deletion can turn into data loss quickly. Experts know which operations are online, which need a maintenance window, and which require a verified backup first.

How it actually works

Common stack:

   physical disk / cloud volume
      -> partition table
      -> LUKS encryption, maybe
      -> RAID or cloud-level replication, maybe
      -> LVM physical volume / volume group / logical volume, maybe
      -> filesystem: ext4, XFS, Btrfs, ZFS, etc.
      -> mount options
      -> application write pattern and fsync behaviour

Safe read-only inventory commands:

lsblk -f
findmnt
df -hT
du -xh /var 2>/dev/null | sort -h | tail
sudo journalctl -p warning -b --no-pager | grep -Ei 'disk|ext4|xfs|nvme|i/o'

Growth workflow, conceptually:

   enlarge underlying disk/volume
   grow partition or PV
   grow LVM logical volume, if used
   grow filesystem
   verify with df/findmnt/application metrics

The exact commands differ by filesystem and cloud provider. Do not paste a resize recipe into production unless you have confirmed the device names, stack layout, filesystem type, and backup state.

Real-world scenario

/var is full because container logs grew without rotation. The wrong fix is blind deletion. The safer flow is:

df -hT
sudo du -xh /var 2>/dev/null | sort -h | tail -20
journalctl --disk-usage
docker system df 2>/dev/null || true

Then decide: rotate/truncate logs, fix log retention, move data, expand the volume, or clean unused container artifacts. The expert result is not "space free right now"; it is "space free, alert threshold corrected, retention fixed, and restore still tested."

Try it yourself

In a disposable VM, attach or create a small extra disk/loop device and practise: identify it with lsblk, create a filesystem, mount it under /mnt/demo, write data, unmount it, and remount it by UUID. Do this only in a lab.

Common mistakes

Confusing RAID, backup, snapshot, and replication. They solve different problems.
Assuming a snapshot is restorable because it exists.
Running cleanup commands before identifying what is consuming space.
Growing storage without fixing the growth rate that filled it.

Check yourself

Why is a tested restore more important than a configured backup job?
What information do you need before resizing a filesystem?
Why can deleting an open log file fail to free space immediately?