Chapter 76 — Evaluating agents: "works once" vs "works every time"

pass@k vs pass^k: the most important idea in this chapter

Run each task n times. Let c be the number of successful runs.

• pass@k = probability that at least one of k attempts succeeds. Useful when a verifier can pick the good attempt (e.g. generate 5 patches, keep the one whose tests pass). It's the number research papers like.
• pass^k ("pass-hat-k", from τ-bench) = probability that all k attempts succeed. This is the reliability number: if 4 customers ask the same thing, do all 4 get it right?

from math import comb


def pass_at_k(n, c, k):
    """P(at least ONE of k tries succeeds), estimated from n tries with c successes.
    Unbiased estimator from the Codex paper (Chen et al., 2021)."""
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)


def pass_hat_k(n, c, k):
    """P(ALL k tries succeed) -- 'pass^k' from tau-bench (Yao et al., 2024).
    This is the reliability number: will it work every time a customer asks?"""
    return comb(c, k) / comb(n, k)


# Results of running each task n=8 times (c = number of successful runs)
results = {"refund_simple": 8, "refund_partial": 7, "address_change": 6,
           "cancel_after_ship": 4, "multi_order_lookup": 5}
n = 8
print(f"{'task':20s} {'pass@1':>7s} {'pass@4':>7s} {'pass^4':>7s}")
for task, c in results.items():
    print(f"{task:20s} {pass_at_k(n, c, 1):7.2f} {pass_at_k(n, c, 4):7.2f} "
          f"{pass_hat_k(n, c, 4):7.2f}")
avg = lambda f, k: sum(f(n, c, k) for c in results.values()) / len(results)
print(f"{'AVERAGE':20s} {avg(pass_at_k, 1):7.2f} {avg(pass_at_k, 4):7.2f} "
      f"{avg(pass_hat_k, 4):7.2f}")

Output:

task                  pass@1  pass@4  pass^4
refund_simple           1.00    1.00    1.00
refund_partial          0.88    1.00    0.50
address_change          0.75    1.00    0.21
cancel_after_ship       0.50    0.99    0.01
multi_order_lookup      0.62    1.00    0.07
AVERAGE                 0.75    1.00    0.36

Read that bottom line twice. The same agent is "100%" (pass@4), "75%" (pass@1), or "36%" (pass^4), depending on which question you ask. Report the one that matches how the agent is used. A support agent that talks to customers directly needs high pass^k. A coding agent whose output goes through tests and code review can live with lower pass^k, because the verifier filters out the bad runs.

What to grade

Grade in layers, from cheapest and most objective to most expensive:

Grade the outcome, not the path, wherever possible. Agents find valid paths you didn't anticipate, and a test that asserts "must call tool A then B then C" fails good agents for being creative. Use trajectory checks for rules (never do X, always do Y before Z, stay under budget), not for the exact route.

Build environments, not just datasets

An agent eval is a small simulation of production: a fresh copy of the world for every run.

   for each task, for each of n trials:
       env   = fresh sandbox: seeded DB, fake payment API, mock email, files
       user  = scripted or SIMULATED user (an LLM playing the customer,
               with a hidden goal and persona -- this is how tau-bench works)
       run the agent in its real harness against env + user
       grade: final env state + trajectory + transcript (judge)
   report: pass^k per task, cost and latency distributions, failure clusters

Two things in that sketch matter more than they look:

• A fresh environment per trial. If trial 3 sees the refund trial 2 issued, your results are garbage. Containers or database snapshots make this cheap.
• The real harness. Evaluate the whole system (prompts, tools, hooks, compaction) not the model in isolation. A harness change can matter as much as a model change (Chapter 74), so it needs the same eval.

Public benchmarks: useful, but not your eval

Use them to shortlist models. Don't use them to decide whether your agent is ready: they aren't your tools, your policies, or your users, and popular benchmarks leak into training data over time (contamination). Your own 50-task eval beats any leaderboard for that decision.

Evals in the development loop

The teams that ship good agents treat evals like tests in CI:

1. Start small and early. Anthropic's write-up of their multi-agent research system noted that in early development, changes have big effects, so even ~20 realistic queries were enough to see whether a prompt change helped. Don't wait for 1,000.
2. Every production failure becomes an eval task. A customer complaint on Tuesday is a regression test by Wednesday.
3. Run the suite on every change to prompt, tools, harness, or model version, and compare against the last good run: pass^k, cost per task, p95 latency, steps per task.
4. Read transcripts. Aggregate scores tell you that something broke; transcripts tell you why. Cluster failures ("12 of 19 failures: called cancel_order on shipped orders") and fix the biggest cluster first.
5. Calibrate the judge. Label 50 transcripts yourself, compare with the LLM judge, and fix the rubric until they agree. Swap answer order to catch position bias (Chapter 49).

Online evals: grading in production

Offline evals cover what you thought of. Production shows you everything else. Online evaluation means continuously grading real traffic:

• run cheap automatic checks on every conversation (policy violations, groundedness, "did the user have to repeat themselves", tool error rate)
• run an LLM judge on a sample (say 2%) and alert when scores drop
• track outcome signals the business already has: refund reversals, escalations to humans, reopened tickets, thumbs-down
• route a sample of flagged conversations to human reviewers, and feed confirmed failures back into the offline suite (step 2 above)

                         old prompt     new prompt
   pass^5 (overall)          0.71          0.64     <-- worse
   avg steps                 5.2           4.1      <-- "better"
   cost / task             $0.031        $0.024     <-- "better"
   forbidden-tool rate       0.0%          1.5%     <-- RED FLAG