62. Experiment hygiene: seeds, configs, notebooks, and reproducibility

62.1 Seeds

import os
import random
import numpy as np

def seed_everything(seed: int) -> None:
    random.seed(seed)
    np.random.seed(seed)                    # legacy global RNG; prefer np.random.default_rng(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)   # only affects subprocesses started after this
    try:
        import torch
        torch.manual_seed(seed)             # CPU and all GPUs
        torch.use_deterministic_algorithms(True, warn_only=True)
    except ImportError:
        pass

seed_everything(42)

Seeds make a run repeatable on the same hardware and library versions. They do not make results trustworthy: report the mean and spread over several seeds, because a single lucky seed can make a worse model look better.

62.2 Configs as frozen dataclasses

import argparse
import json
from dataclasses import dataclass, asdict, fields

@dataclass(frozen=True)
class Config:
    lr: float = 3e-4
    batch_size: int = 64
    epochs: int = 10
    seed: int = 42
    data: str = "corpus.jsonl"

def parse() -> Config:
    p = argparse.ArgumentParser()
    for f in fields(Config):
        p.add_argument(f"--{f.name.replace('_', '-')}", type=type(f.default), default=f.default)
    return Config(**vars(p.parse_args([])))     # pass sys.argv[1:] in real use

cfg = parse()
print(json.dumps(asdict(cfg)))                  # save next to every run's outputs

Every run should write its config, the git commit (git rev-parse HEAD), the uv.lock hash, and its metrics into its own output directory. That is what lets you answer "which settings produced this model?" three months later. Experiment trackers (MLflow, Weights & Biases, TensorBoard) automate the bookkeeping.

62.3 Notebooks: for exploring, not for shipping

Jupyter notebooks are excellent for exploration: load data, plot it, try an idea, see the result inline. They are poor as the source of truth: cells run out of order, hidden state survives deleted cells, diffs are JSON noise, and outputs (sometimes including secrets) are saved in the file.

The workflow that works: explore in a notebook; move anything you will run twice into a module under src/ with tests; import that module back into the notebook. "Restart kernel and run all" must succeed before you trust a notebook's result. jupytext stores notebooks as .py files for clean diffs, and nbstripout removes outputs on commit.

62.4 Evaluation sets are code

For any model, and especially for LLM apps, keep a versioned file of inputs with expected outputs (or grading criteria), and a script that scores the current system against it. Run it on every change, like the tests in Ch 20. Without it, every prompt or retrieval tweak is a guess. The AI guide Ch 49 shows how to build one.