34. Processes: subprocess done right, exit codes, and signals

34.1 Running commands

import subprocess

r = subprocess.run(
    ["ls", "-l", "/etc/hostname"],   # a LIST of arguments: no shell involved
    capture_output=True,
    text=True,                       # decode stdout/stderr as str
    timeout=10,                      # never wait forever
    check=True,                      # raise CalledProcessError on non-zero exit
)
print(r.returncode, r.stdout.strip())

subprocess.run does fork + exec (or posix_spawn, a faster path CPython uses when it can) and waits, as in OS Ch 6.

Never use shell=True with any input you did not write. It passes the string to /bin/sh, so shell metacharacters in the input become commands:

import subprocess
filename = "notes.txt; rm -rf ~"          # attacker-controlled
# WRONG: subprocess.run(f"cat {filename}", shell=True)  -> runs `rm -rf ~`
subprocess.run(["cat", filename], check=False)   # safe: one argument, no shell

That is OS command injection, the same class as SQL injection in the Security guide Ch 42. If you truly need shell features (pipes, globs), build them in Python, or quote with shlex.quote and keep the input on an allow-list.

Streaming output from a long-running command:

import subprocess

with subprocess.Popen(["ping", "-c", "3", "127.0.0.1"], stdout=subprocess.PIPE, text=True) as p:
    for line in p.stdout:              # line by line as it arrives
        print("ping>", line.rstrip())
print("exit", p.returncode)

Exit codes follow the shell convention: 0 success, 1–125 errors, and a process killed by signal N shows up as returncode == -N in Python (the shell shows 128 + N).

34.2 Signals and graceful shutdown

Kubernetes, systemd, and Docker stop a process by sending SIGTERM, waiting a grace period, then sending SIGKILL (OS Ch 20). A worker that ignores SIGTERM loses its in-flight work every deploy.

# labs/worker.py: finish the current job, then exit cleanly on SIGTERM / Ctrl-C
import os
import signal
import threading
import time

stop = threading.Event()

def on_signal(signum, frame):
    print(f"got {signal.Signals(signum).name}; finishing current job", flush=True)
    stop.set()

signal.signal(signal.SIGTERM, on_signal)
signal.signal(signal.SIGINT, on_signal)

print(f"worker pid={os.getpid()}", flush=True)
job = 0
while not stop.is_set():
    job += 1
    print(f"job {job} start", flush=True)
    stop.wait(timeout=2.0)            # stand-in for real work; wakes early on stop
    print(f"job {job} done", flush=True)
print("clean exit")

python3 labs/worker.py & sleep 3; kill -TERM %1; wait

Python-specific rules:

• Python runs signal handlers in the main thread, between bytecode instructions. A handler cannot interrupt a long C call (a big NumPy operation) until it returns. Keep handlers tiny: set a flag or an Event, and let the main loop act on it.
• Ctrl-C raises KeyboardInterrupt by default; that is a SIGINT handler Python installs for you.
• asyncio has its own API: loop.add_signal_handler(signal.SIGTERM, cb).

34.3 PID 1 in containers

If your Python program is PID 1 in a container (CMD ["python", "app.py"] with exec form), the kernel applies no default signal actions to it: SIGTERM is ignored unless you install a handler, and orphaned child processes are never reaped and pile up as zombies. Either handle SIGTERM explicitly, or run a tiny init as PID 1 (docker run --init, or tini) so signals are forwarded and zombies reaped. The OS guide Ch 74 has the full story with Go; the rules are identical for Python.

34.4 Project: a tiny supervisor

# labs/supervisor.py: run a command, restart it on crash with backoff, forward SIGTERM
import signal
import subprocess
import sys
import time

def main(cmd: list[str]) -> int:
    child: subprocess.Popen | None = None
    stopping = False

    def forward(signum, frame):
        nonlocal stopping
        stopping = True
        if child and child.poll() is None:
            child.send_signal(signum)

    signal.signal(signal.SIGTERM, forward)
    signal.signal(signal.SIGINT, forward)

    backoff = 0.5
    while True:
        started = time.monotonic()
        child = subprocess.Popen(cmd)
        code = child.wait()
        if stopping:
            print(f"supervisor: child exited {code} after stop request", file=sys.stderr)
            return 0
        ran = time.monotonic() - started
        backoff = 0.5 if ran > 10 else min(backoff * 2, 30)   # reset after a healthy run
        print(f"supervisor: child exited {code}; restarting in {backoff:.1f}s", file=sys.stderr)
        time.sleep(backoff)

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1:] or ["python3", "-c", "import time; time.sleep(1); raise SystemExit(3)"]))

This is a miniature systemd Restart=on-failure with RestartSec backoff (OS Ch 33). In production, let systemd or Kubernetes supervise; write this once to understand what they do.