37. mmap, shared memory, and pipes

37.1 mmap: a file as memory

import mmap
import os

path = "/tmp/big.bin"
with open(path, "wb") as f:
    f.truncate(64 * 2**20)                    # a 64 MiB sparse file

with open(path, "r+b") as f, mmap.mmap(f.fileno(), 0) as m:
    m[0:5] = b"HELLO"                         # writes go to the page cache directly
    m[-5:] = b"WORLD"
    print(m[:5], m[-5:], len(m))
    print(m.find(b"WORLD"))                   # search without reading the file into Python
os.remove(path)

mmap maps the file into your address space, so pages are loaded on first touch (OS Ch 11) and shared between processes. NumPy's np.memmap, Hugging Face safetensors, and llama.cpp's GGUF loading all use mmap to "load" a multi-gigabyte model in milliseconds: nothing is read until a weight is used, and several processes share one copy in the page cache.

37.2 Shared memory between processes

# labs/shm.py: workers fill one shared NumPy array; nothing is pickled or copied
import numpy as np
from multiprocessing import Process, shared_memory

def worker(name: str, shape: tuple[int, ...], row: int) -> None:
    shm = shared_memory.SharedMemory(name=name)
    arr = np.ndarray(shape, dtype=np.float64, buffer=shm.buf)
    arr[row, :] = row * 10.0                       # writes directly into shared pages
    del arr                                        # drop the view before closing
    shm.close()

if __name__ == "__main__":
    shape = (4, 1_000_000)                         # 32 MB
    shm = shared_memory.SharedMemory(create=True, size=int(np.prod(shape)) * 8)
    try:
        arr = np.ndarray(shape, dtype=np.float64, buffer=shm.buf)
        ps = [Process(target=worker, args=(shm.name, shape, r)) for r in range(shape[0])]
        for p in ps: p.start()
        for p in ps: p.join()
        print(arr[:, 0], arr.sum())
        del arr
    finally:
        shm.close()
        shm.unlink()                               # remove /dev/shm/<name>

This is the fix for the "processes copy my data" problem from Ch 28, and it is what PyTorch's DataLoader uses to hand batches back from worker processes (/dev/shm, which is why Docker's default 64 MB /dev/shm breaks it: run with --shm-size=2g or --ipc=host).

37.3 Pipes

subprocess.PIPE, os.pipe(), and multiprocessing.Pipe are all the kernel pipe from OS Ch 19. A classic deadlock: reading a child's stdout to the end while it blocks writing to a full stderr pipe that you never read. Popen.communicate() reads both concurrently; use it (or run(capture_output=True)) instead of reading the pipes one after the other.