32. Project: a parallel file hasher, four ways

Goal: compute SHA-256 hashes of every file under a directory (a dataset, a model directory, a backup) and compare four implementations: serial, threads, processes, and asyncio. Predict the winner before you run it, then explain the result with Chapters 27–29.

# labs/hasher.py
import asyncio
import hashlib
import os
import sys
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from pathlib import Path

CHUNK = 1 << 20   # 1 MiB reads: big enough to amortize syscalls, small enough for constant memory

def sha256_file(path: Path) -> tuple[str, str]:
    h = hashlib.sha256()
    with path.open("rb") as f:
        while chunk := f.read(CHUNK):
            h.update(chunk)            # releases the GIL for large chunks
    return str(path), h.hexdigest()

def files_under(root: Path) -> list[Path]:
    return [p for p in root.rglob("*") if p.is_file()]

def serial(files):
    return dict(map(sha256_file, files))

def threaded(files):
    with ThreadPoolExecutor(max_workers=os.cpu_count()) as pool:
        return dict(pool.map(sha256_file, files))

def processes(files):
    with ProcessPoolExecutor() as pool:
        return dict(pool.map(sha256_file, files, chunksize=8))

def with_asyncio(files):
    async def main():
        sem = asyncio.Semaphore(os.cpu_count() or 4)
        async def one(p):
            async with sem:
                return await asyncio.to_thread(sha256_file, p)
        return dict(await asyncio.gather(*(one(p) for p in files)))
    return asyncio.run(main())

def make_dataset(root: Path, n: int = 64, size: int = 8 << 20) -> None:
    root.mkdir(exist_ok=True)
    for i in range(n):
        p = root / f"shard-{i:03}.bin"
        if not p.exists():
            p.write_bytes(os.urandom(size))

if __name__ == "__main__":
    root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("hash-data")
    if len(sys.argv) == 1:
        make_dataset(root)                       # 64 x 8 MiB = 512 MiB of random data
    files = files_under(root)
    reference = None
    for fn in (serial, threaded, processes, with_asyncio):
        t = time.perf_counter()
        out = fn(files)
        dt = time.perf_counter() - t
        reference = reference or out
        assert out == reference
        mb = sum(p.stat().st_size for p in files) / 2**20
        print(f"{fn.__name__:12} {dt:6.2f}s  {mb / dt:7.0f} MiB/s")

Measured while this guide was written, on a 14-core Apple Silicon laptop with the files already in the page cache (run it twice; the first run reads from disk):

serial         0.23s     2215 MiB/s
threaded       0.02s    20640 MiB/s
processes      0.11s     4690 MiB/s
with_asyncio   0.03s    17536 MiB/s

Why threads win here: hashlib releases the GIL while hashing large chunks, and file reads release it too, so threads run in parallel with no pickling and no process start-up. Processes pay to start workers. asyncio only adds scheduling on top of threads, because there is no async file I/O in the standard library. Change sha256_file to a pure-Python checksum (sum(chunk) % 2**32) and the ranking flips: processes win and threads lose. On a free-threaded 3.14t build, threads win both versions.

Then run it on a dataset that does not fit in the page cache, or on a network filesystem, and watch every version converge to the disk's speed. The OS guide Ch 14 and 16 explain why: once you are I/O-bound, more CPUs do nothing.