28. Processes: multiprocessing and concurrent.futures

For CPU-bound pure-Python work on the default build, use several processes. Each has its own interpreter and its own GIL, so they run on different cores.

import math
import time
from concurrent.futures import ProcessPoolExecutor

def count_primes(limit: int) -> int:
    count = 0
    for n in range(2, limit):
        if all(n % d for d in range(2, math.isqrt(n) + 1)):
            count += 1
    return count

if __name__ == "__main__":                     # REQUIRED with process pools (see below)
    chunks = [200_000] * 8
    t = time.perf_counter()
    serial = [count_primes(c) for c in chunks]
    t_serial = time.perf_counter() - t

    t = time.perf_counter()
    with ProcessPoolExecutor() as pool:        # default: one worker per CPU
        parallel = list(pool.map(count_primes, chunks))
    t_par = time.perf_counter() - t
    assert serial == parallel
    print(f"serial {t_serial:.2f}s  processes {t_par:.2f}s  speedup {t_serial / t_par:.1f}x")

What you pay for processes:

• Start-up cost. Each worker is a new process. On macOS and Windows the default start method is spawn (a fresh interpreter that re-imports your main module); on Linux, 3.14 changed the default from fork to forkserver, because forking a process that has threads is unsafe. This is why the if __name__ == "__main__": guard is required: without it, every worker would re-run the pool creation on import.
• Serialization cost. Arguments and results are pickled, sent over a pipe, and unpickled. Sending a 1 GB array to each worker copies 1 GB each time. Send small task descriptions (file paths, index ranges), and let workers load their own data; or use shared memory (Ch 37).
• Lambdas and locally defined functions cannot be pickled; worker functions must be importable top-level functions.

Real-world example: PyTorch's DataLoader(num_workers=8) is exactly this: eight worker processes decode and augment images in parallel and send finished batches back to the training process through shared memory. When it hangs on macOS or crashes with a pickling error, the cause is almost always one of the three points above.

28.1 3.14: subinterpreters

3.14 added concurrent.futures.InterpreterPoolExecutor (PEP 734): several interpreters in one process, each with its own GIL. Start-up is cheaper than processes and they run in parallel, but objects still cannot be shared directly; data is passed by copying. It is new and few C extensions support it yet. Watch it, but use processes in production today.

28.2 Choosing a concurrency model