Chapter 79 — Project 6: an overnight agent that opens real pull requests

Step 1 — Project rules and the skill

AGENTS.md (in the repo, used by humans' agents too):

## Test
- `make lint typecheck test` must all pass. That is the definition of done.
## Rules
- Never modify files under tests/ to make a failing test pass. If a test is
  genuinely wrong because the library changed behaviour, STOP and explain in
  progress.md instead.
- Never edit .github/ or Makefile.

skills/dependency-upgrade/SKILL.md:

---
name: dependency-upgrade
description: Upgrade one dependency and fix breakages. Use for any task that
  says "upgrade <package>".
---
1. Read the package's changelog / release notes between the current and
   target version (the URL is in the task). List breaking changes first.
2. Bump ONLY that package (and its required peers) in the lockfile.
3. Run `make test`. For each failure, find the changelog entry that explains
   it, and fix OUR code to match the new API. Prefer the library's
   recommended migration over workarounds.
4. Search for usages of any deprecated API mentioned in the changelog, even
   if tests pass, and migrate them.
5. In progress.md write: breaking changes found, files changed, anything a
   reviewer should look at closely, and your risk rating (low/med/high) with
   one sentence of reasoning.

Step 2 — Plan: turn outdated dependencies into tasks

# plan.py -- one task per outdated dependency, patch/minor before major
import json, subprocess

out = subprocess.run(["pip", "list", "--outdated", "--format=json"],
                     capture_output=True, text=True, check=True).stdout
deps = json.loads(out)

def risk(d):                      # semver distance: patch < minor < major
    cur, new = d["version"].split("."), d["latest_version"].split(".")
    return 2 if cur[0] != new[0] else 1 if cur[1:2] != new[1:2] else 0

tasks = []
for d in sorted(deps, key=risk)[:8]:                  # at most 8 a night
    tasks.append({
        "id": f"deps-{d['name']}-{d['latest_version']}",
        "desc": (f"upgrade {d['name']} from {d['version']} to "
                 f"{d['latest_version']} (use the dependency-upgrade skill; "
                 f"changelog: https://pypi.org/project/{d['name']}/)"),
        "verify": "make lint typecheck test && python check_tests_untouched.py",
        "passes": False,
    })
json.dump(tasks, open("features.json", "w"), indent=2)
print(f"{len(tasks)} tasks planned")

(Use your ecosystem's equivalent: npm outdated --json, go list -m -u -json all, cargo outdated, and so on.)

Step 3 — The extra verifier: tests must not be weakened

The most common cheat in long-running coding agents is making the tests easier. Make it impossible to get a PASS that way:

# check_tests_untouched.py -- fail if this attempt weakened the test suite
import re, subprocess, sys

# compare against main, so changes the agent already COMMITTED are included
diff = subprocess.run(["git", "diff", "main", "--", "tests/"],
                      capture_output=True, text=True).stdout
deleted_tests = re.findall(r"^-\s*def test_", diff, re.M)
new_skips = re.findall(r"^\+.*(@pytest\.mark\.(skip|xfail)|pytest\.skip\()", diff, re.M)
if deleted_tests or new_skips:
    print(f"tests weakened: {len(deleted_tests)} removed, {len(new_skips)} skips added")
    sys.exit(1)

Step 4 — Run the outer loop in the sandbox

Reuse outer_loop.py from Chapter 75, with features.json, progress.md, and STOP in .gitignore, and one addition. Before each task, create the branch, so each passing task's harness commit lands on its own branch:

# in main(), right after picking `task`:
sh(f"git checkout -q main && git checkout -q -B {task['id']}")

Set the budgets for the night: MAX_ITERATIONS = 24 (8 tasks × 3 attempts), MAX_HOURS = 3, and a provider-side spend limit on the API key as a second fence.

Step 5 — Open the PRs (the human gate)

# open_prs.py -- one PR per verified branch; humans decide
import json, subprocess

for t in json.load(open("features.json")):
    if not t["passes"]:
        continue
    notes = subprocess.run(["grep", t["id"], "progress.md"],
                           capture_output=True, text=True).stdout
    body = (f"Automated upgrade, verified by `{t['verify']}`.\n\n"
            f"### Agent notes\n{notes}\n\n"
            f"Review the risk rating and changed files before merging.")
    subprocess.run(["git", "push", "-q", "origin", t["id"]], check=True)
    subprocess.run(["gh", "pr", "create", "--head", t["id"], "--base", "main",
                    "--title", t["desc"].split(" (")[0],
                    "--body", body, "--label", "agent"], check=True)

Note that the harness pushes, not the agent. The agent's own permissions deny git push (Chapter 74); only this small, deterministic script, which pushes branches the harness verified, has the token.

Step 6 — Schedule it (CI as the runtime)

Any CI system with a scheduler works. A GitHub Actions sketch:

# .github/workflows/nightly-deps-agent.yml
name: nightly-deps-agent
on:
  schedule: [{cron: "0 2 * * 1-5"}]     # 02:00 UTC, weekdays
  workflow_dispatch: {}                  # manual run button
concurrency: deps-agent                  # never two runs at once
jobs:
  run:
    runs-on: ubuntu-latest
    timeout-minutes: 200                 # hard wall-clock limit
    permissions: {contents: write, pull-requests: write}   # nothing else
    steps:
      - uses: actions/checkout@v4
        with: {fetch-depth: 0}
      - run: pip install -r requirements-dev.txt
      - run: npm install -g @anthropic-ai/claude-code     # or your agent CLI
      - run: python plan.py
      - run: python outer_loop.py claude -p
        env:
          ANTHROPIC_API_KEY: ${{ secrets.DEPS_AGENT_API_KEY }}   # spend-capped key
      - run: python open_prs.py
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - uses: actions/upload-artifact@v4                  # the morning report
        with: {name: agent-logs, path: "progress.md\nfeatures.json"}

The CI runner is the sandbox here: a fresh VM per run, destroyed afterwards, with exactly two secrets. For stricter isolation (egress allowlists, microVMs), run the same scripts on your own runners (Chapter 77).

Step 7 — Evaluate before you trust it

Before turning on the schedule, build an eval set from history (Chapter 76): find 15 past dependency upgrades in your git log that needed code changes. For each one, check out the commit before the upgrade, run the agent on that single task 3 times, and grade:

• outcome: does the full suite pass, and do the hidden tests that the human added in the real upgrade commit pass?
• rules: no test files weakened, no files outside the allowed paths changed
• cost and steps per task

You'll get a pass^3 per upgrade type. Typical findings: patch and minor bumps pass reliably; major versions with big API changes fail often. That's fine, because those become "needs human" in the morning report, which is still useful: the agent has already read the changelog and listed the breaking changes.

Step 8 — Watch it in production

Each morning's summary (posted to team chat by a final CI step) should show:

   Nightly deps agent -- 2026-10-03
   tasks: 8   verified PRs: 5   needs human: 2   skipped (budget): 1
   cost: $6.40 (limit $15)   wall time: 1h52m
   PRs: #4190 requests 2.32->2.33 (low)   #4191 pydantic 2.9->2.11 (med) ...
   needs human: sqlalchemy 1.4->2.0 (major; 41 call sites use Query API,
                see progress.md)

Track over weeks: PR merge rate (the true success metric), time reviewers spend per PR, cost per merged PR, and reverted agent PRs (should be ~0). If reviewers start rubber-stamping, slow the agent down; the human gate only works if humans actually review.