35. Files that survive crashes: fsync and atomic replacement

Writing a config file, a checkpoint, or a state file with open(path, "w").write(data) has two failure modes:

1. A crash midway leaves a truncated file ("w" truncates first).
2. A power loss after write returns can leave the file empty or old, because the data was only in the page cache.

The standard fix, from the OS guide Ch 77: write a temporary file in the same directory, fsync it, rename it over the target (atomic on POSIX filesystems), then fsync the directory so the rename itself is durable.

# labs/atomicwrite.py
import os
import tempfile
from pathlib import Path

def atomic_write(path: Path, data: bytes) -> None:
    path = Path(path)
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f".{path.name}.", suffix=".tmp")
    try:
        with os.fdopen(fd, "wb") as f:
            f.write(data)
            f.flush()                    # Python buffer -> kernel
            os.fsync(f.fileno())         # kernel page cache -> disk
        os.replace(tmp, path)            # atomic rename(2), overwrites on every OS
    except BaseException:
        os.unlink(tmp)
        raise
    dfd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(dfd)                    # make the rename durable
    finally:
        os.close(dfd)

if __name__ == "__main__":
    target = Path("/tmp/state.json")
    atomic_write(target, b'{"step": 1200, "loss": 2.31}\n')
    print(target.read_text())

Readers now see either the complete old file or the complete new one, never a half-written one.

Real-world example: training checkpoints. torch.save(state, "ckpt.pt") writes in place. If the job is preempted (spot instances, a Kubernetes eviction) during the save, you lose the latest checkpoint and the previous one, because it was overwritten. Save to ckpt.pt.tmp, then os.replace. Most training frameworks do this internally now; check that yours does.