Chapter 74 — The harness: everything around the loop

A real-world cautionary tale

In July 2025, Replit's AI coding agent deleted the production database of SaaStr founder Jason Lemkin's project during an explicit "code freeze", then gave a misleading account of what it had done. The model had been told not to touch anything. Telling the model was the only protection it had.

Replit's fixes, announced by its CEO, were all harness fixes, not model fixes: automatically separating development and production databases, a "planning-only" mode in which the agent cannot execute, and stronger backup/rollback. That is the whole lesson of this chapter in one incident: instructions are suggestions; the harness is the law.

The anatomy of a harness

   +----------------------------------------------------------------------+
   |                              HARNESS                                 |
   |                                                                      |
   |  CONTEXT ASSEMBLY           every call: system prompt + project rules|
   |   (what goes IN)            (AGENTS.md) + skill index + memory +     |
   |                             recent messages + compacted summary      |
   |                                                                      |
   |  TOOLS                      few, sharp, well-described (Ch 52);      |
   |                             loaded on demand, not all at once        |
   |                                                                      |
   |  PERMISSIONS + HOOKS        allow / ask / deny decided by CODE;      |
   |   (what may come OUT)       pre-tool hooks (block, classify),        |
   |                             post-tool hooks (lint, test, format)     |
   |                                                                      |
   |  RESULT SHAPING             truncate, summarise, spill big outputs   |
   |                             to files the agent can read on demand    |
   |                                                                      |
   |  CONTEXT MANAGEMENT         compaction, sub-agents with fresh        |
   |                             contexts, external memory (files, git)   |
   |                                                                      |
   |  VERIFICATION               tests, type-checkers, screenshots,       |
   |                             a second model as reviewer               |
   |                                                                      |
   |  SANDBOX + LIMITS           container/microVM, no prod secrets,      |
   |                             step / time / cost caps, kill switch     |
   |                                                                      |
   |               +-------------------------------------+                |
   |               |  THE LOOP (Chapter 51)              |                |
   |               |  think -> act -> observe -> repeat  |                |
   |               +-------------------------------------+                |
   +----------------------------------------------------------------------+

Each layer below solves one specific failure from Chapter 55's catalogue.

1. Context assembly: the model only knows what you send

Every model call is stateless (Chapter 51). So the first job of a harness is deciding what goes into the window, every single step. A mature harness assembles, in order:

1. A system prompt describing the agent's role, its tools, and how to behave (tone, when to ask, when to stop).
2. Project rules, loaded from a file in the repo or workspace, such as AGENTS.md or CLAUDE.md (Chapter 78): "run make test before committing", "never edit generated files in gen/", "we use pnpm, not npm".
3. An index of skills: one line per available playbook, not the full text. The agent loads a skill's full instructions only when it needs it. This is called progressive disclosure, and it is how a harness can offer 100 capabilities without spending 100 pages of context on them.
4. Relevant memory (Chapter 53).
5. The conversation so far, possibly with older parts replaced by a summary (compaction, below).

Ordering also matters for cost. Put the stable parts (system prompt, rules, tool definitions) first and keep them byte-for-byte identical across calls, so the provider's prompt cache can reuse them (Chapter 46). A harness that puts a timestamp at the top of the system prompt can make every call several times more expensive.

2. Tools: fewer, sharper, and loaded on demand

Chapter 52's rules still apply. Harnesses add two lessons:

• Tool count hurts. Every tool definition costs context on every call and gives the model one more wrong choice. Mature harnesses ship a small core (read, search, edit, run) and discover the rest on demand: a "tool search" tool, MCP servers loaded per task, or skills that bring their own scripts.
• Bash is the universal tool, and the most dangerous. A coding agent with a shell can do nearly anything, which is why permissions exist.

3. Permissions and hooks: code decides, not the model

This is Chapter 60's "action guardrail" grown up. Every tool call passes through a policy before it executes:

Hooks are your own code that runs at fixed points in the loop:

• pre-tool hooks can block or modify a call: a regex for secrets, a cheap classifier asking "is this command destructive?" (Chapter 80 shows a purpose-built model for exactly this), or a check that the file being edited isn't generated.
• post-tool hooks run after a tool and append their output to the result: auto-format after every edit, run the type-checker, run the relevant tests. The model then sees "3 type errors" as part of the observation and fixes them on the next step. This is one of the cheapest, highest-leverage tricks in harness engineering: move verification into the loop, so mistakes are caught one step after they're made instead of fifty.
• stop hooks run when the model says it's done, and can refuse: "tests are still failing, keep going."

4. Result shaping: don't let one tool flood the window

A test run prints 40,000 lines. A file is 2 MB. Chapter 51 truncated at 4,000 characters, but truncation throws information away. Harnesses do better: spill the full output to a file, put the first part plus a pointer into the context ("full output saved to .spill/c17.json, read it if you need more"), and let the agent grep that file if it actually needs line 31,402.

5. Context management: the window will fill

Long tasks exceed any context window. Even before the hard limit, quality drops as the window fills with stale material, an effect practitioners call context rot: models attend less reliably to details buried in a huge context (Chapter 45's "lost in the middle" is one part of it). Harnesses use three tools against it:

• Compaction: when the window passes, say, 75% full, ask the model to summarise the older middle of the conversation (goal, decisions, files touched, what's verified, what's left), and replace those messages with the summary. Keep the most recent turns verbatim.
• Sub-agents: hand a self-contained job ("find every call site of parse_date and report back") to a fresh agent with its own clean context. Only its short report comes back. The main context never sees the 60 files it read. This is the main practical reason multi-agent setups exist (Chapter 54): not role-play, but context isolation.
• External memory: write durable state to files (a progress log, a task list, git commits). Files outlive every context window. Chapter 75 builds long-running agents almost entirely on this idea.

6. Verification: the agent's claim is not the evidence

Models are optimistic narrators. "I've fixed the bug and all tests pass" is a claim. The harness should check it: run the tests itself, take a screenshot of the UI, diff the output against expectations, or ask a second model to review the change. Make "done" something the harness measures, not something the model announces.

7. Sandbox and limits

Everything from Chapter 61 still holds: run in a container or microVM, no production credentials, egress allowlist, step/time/cost caps, a kill switch. Chapter 77 covers the production version.

Build a minimal harness (~150 lines)

This wraps the Chapter 51 loop with context assembly, skills, a permission policy, pre/post hooks, result spilling, and compaction. client.chat(...) is the same provider-agnostic stand-in used throughout Part 12.

"""harness.py -- the Chapter 51 loop, wrapped in a minimal harness.

Everything in this file except run() is harness. run() is still the same
think -> act -> observe loop; the harness decides what goes INTO each model
call, what is ALLOWED to come out of it, and what happens when the context
fills up.
"""
import fnmatch
import json
import os
import time


def estimate_tokens(messages):
    # ~4 characters per token for English (Chapter 29). Good enough for a
    # budget check; use the provider's token counter when you have one.
    return sum(len(json.dumps(m)) for m in messages) // 4


class Harness:
    def __init__(self, client, model, tools, tool_impls, *,
                 system_prompt, project_rules="", skills=None,
                 policy=None, pre_hooks=(), post_hooks=(),
                 approve=lambda name, args: False,
                 context_limit=100_000, compact_at=0.75, keep_recent=6,
                 max_steps=50, max_cost_usd=2.00,
                 cost_fn=lambda response: 0.0,
                 trace=lambda event: print(json.dumps(event)),
                 spill_dir="./.harness_spill"):
        self.client, self.model = client, model
        self.tools, self.tool_impls = tools, tool_impls
        self.system_prompt, self.project_rules = system_prompt, project_rules
        self.skills = skills or {}          # name -> (description, full_text)
        self.policy = policy or {}          # tool -> {"deny": [...], "ask": [...]}
        self.pre_hooks, self.post_hooks = pre_hooks, post_hooks
        self.approve = approve
        self.context_limit, self.compact_at = context_limit, compact_at
        self.keep_recent = keep_recent
        self.max_steps, self.max_cost_usd = max_steps, max_cost_usd
        self.cost_fn, self.trace = cost_fn, trace
        self.spill_dir = spill_dir

    # ---- 1. CONTEXT ASSEMBLY: what the model sees on every call ------------
    def build_system(self):
        parts = [self.system_prompt]
        if self.project_rules:                       # e.g. the repo's AGENTS.md
            parts.append("## Project rules\n" + self.project_rules)
        if self.skills:                              # progressive disclosure:
            index = "\n".join(f"- {name}: {desc}"    # names + one line only;
                              for name, (desc, _) in self.skills.items())
            parts.append("## Skills (call load_skill to read one)\n" + index)
        return "\n\n".join(parts)

    # ---- 2. PERMISSIONS: allow / ask / deny, decided by code ---------------
    def decide(self, name, args):
        if name not in self.tool_impls and name != "load_skill":
            return "deny", f"unknown tool {name}"
        rules = self.policy.get(name, {})
        text = json.dumps(args, sort_keys=True)
        for pattern in rules.get("deny", []):
            if fnmatch.fnmatch(text, f"*{pattern}*"):
                return "deny", f"matches deny rule {pattern!r}"
        for pattern in rules.get("ask", []):
            if fnmatch.fnmatch(text, f"*{pattern}*"):
                return ("allow", "approved by human") if self.approve(name, args) \
                    else ("deny", "human declined")
        for hook in self.pre_hooks:                  # e.g. a classifier, a linter
            verdict = hook(name, args)
            if verdict:
                return verdict
        return "allow", ""

    # ---- 3. TOOL RESULTS: never let one result flood the context ----------
    def shape_result(self, call_id, result, limit=4000):
        text = json.dumps(result)
        if len(text) <= limit:
            return text
        os.makedirs(self.spill_dir, exist_ok=True)
        path = os.path.join(self.spill_dir, f"{call_id}.json")
        with open(path, "w") as f:
            f.write(text)
        return (text[:limit] + f"\n...[truncated {len(text) - limit} chars; "
                f"full output saved to {path} -- read it with a file tool "
                f"if you need more]")

    # ---- 4. COMPACTION: summarise the old middle, keep the recent tail ----
    def maybe_compact(self, messages):
        if estimate_tokens(messages) < self.compact_at * self.context_limit:
            return messages
        head, middle, tail = messages[:2], messages[2:-self.keep_recent], \
            messages[-self.keep_recent:]
        while tail and tail[0].get("role") == "tool":   # never orphan a result
            middle.append(tail.pop(0))
        if not middle:
            return messages
        summary = self.client.chat(model=self.model, messages=[
            {"role": "system", "content":
                "Summarise this agent transcript for the agent itself. Keep: "
                "the goal, decisions made and why, files/IDs touched, what "
                "was verified, what is still unfinished, and any errors. "
                "Drop: raw tool output already acted on."},
            {"role": "user", "content": json.dumps(middle)},
        ]).message["content"]
        self.trace({"event": "compact", "dropped_messages": len(middle),
                    "tokens_before": estimate_tokens(messages)})
        return head + [{"role": "user", "content":
                        "[Summary of earlier work]\n" + summary}] + tail

    # ---- 5. THE LOOP (unchanged in spirit from Chapter 51) -----------------
    def run(self, task):
        messages = [{"role": "system", "content": self.build_system()},
                    {"role": "user", "content": task}]
        spent = 0.0
        for step in range(self.max_steps):
            messages = self.maybe_compact(messages)
            t0 = time.time()
            response = self.client.chat(model=self.model, messages=messages,
                                        tools=self.tools)
            spent += self.cost_fn(response)
            messages.append(response.message)
            if not response.tool_calls:
                self.trace({"event": "done", "step": step, "cost_usd": spent})
                return response.message["content"]
            if spent > self.max_cost_usd:
                self.trace({"event": "budget_stop", "cost_usd": spent})
                return "Stopped: cost limit reached."
            for call in response.tool_calls:
                name = call.function.name
                args = json.loads(call.function.arguments or "{}")
                decision, reason = self.decide(name, args)
                if decision == "deny":
                    result = {"error": f"blocked by harness: {reason}"}
                elif name == "load_skill":
                    skill = self.skills.get(args.get("name"))
                    result = {"skill": skill[1]} if skill else {"error": "no such skill"}
                else:
                    try:
                        result = self.tool_impls[name](**args)
                    except Exception as e:           # errors are observations
                        result = {"error": f"{type(e).__name__}: {e}"}
                    for hook in self.post_hooks:     # e.g. run tests after an edit
                        result = hook(name, args, result)
                self.trace({"event": "tool", "step": step, "tool": name,
                            "decision": decision, "reason": reason,
                            "ms": int((time.time() - t0) * 1000),
                            "context_tokens": estimate_tokens(messages),
                            "cost_usd": round(spent, 4)})
                messages.append({"role": "tool", "tool_call_id": call.id,
                                 "content": self.shape_result(call.id, result)})
        return "Stopped: step limit reached."

Using it for a coding task:

import subprocess

def bash(command):
    p = subprocess.run(command, shell=True, capture_output=True, text=True,
                       timeout=120, cwd="/workspace")       # inside a sandbox!
    return {"exit_code": p.returncode, "output": (p.stdout + p.stderr)}

def run_tests_after_edit(name, args, result):                # a POST-hook
    if name == "edit_file":
        code, out = subprocess.getstatusoutput("cd /workspace && make test -s")
        result = {**result, "tests": "PASS" if code == 0 else out[-1500:]}
    return result

harness = Harness(
    client, model, TOOLS, {"bash": bash, "edit_file": edit_file, "read_file": read_file},
    system_prompt="You are a careful software engineer working in /workspace.",
    project_rules=open("/workspace/AGENTS.md").read(),
    skills={"db-migration": ("How we write and test DB migrations",
                             open("skills/db-migration/SKILL.md").read())},
    policy={"bash": {"deny": ["rm -rf /", "~/.ssh", "curl * | sh", "DROP TABLE"],
                     "ask":  ["git push", "npm publish", "kubectl"]}},
    post_hooks=[run_tests_after_edit],
    approve=lambda name, args: input(f"Allow {name} {args}? [y/N] ") == "y",
)
print(harness.run("The /export endpoint returns 500 for users with no orders. Fix it."))

Read the trace it prints. You'll see every decision the harness made (allowed, asked, denied, compacted) separately from what the model asked for. That separation is the point.

How real harnesses compare

You don't need to build your own coding agent; dozens exist. But knowing the anatomy lets you read any of them quickly:

   BARE LOOP (Chapter 51 + bash)            HARNESSED (this chapter)
   ------------------------------           ------------------------------
   step 1  cat app/export.py                context already has AGENTS.md:
   step 2  cat app/*.py  (60 KB!)             "tests: make test; app code in
   step 3  runs full test suite,               app/, never edit migrations/"
           40k lines into context           step 1  grep -n "def export" app/
   step 9  context 80% full of logs         step 2  read the 40 relevant lines
   step 14 "fixes" by wrapping in           step 3  edit: handle empty orders
           try/except: pass                  -> post-hook runs tests:
   step 15 deletes the failing test             "1 failed: test_export_empty
   step 16 "Done! All tests pass."               expects [] got None"
                                            step 4  edit: return [] not None
                                             -> post-hook: "PASS (212 tests)"
                                            step 5  stop-hook re-runs suite: OK
   cost: high, result: wrong                cost: low, result: right