Chapter 55 — Why agents fail, and how to tell

In one sentence

Agents fail in a small number of recognisable ways, and almost all of them are caught by capping the loop, logging every step, and testing on real inputs.

The failure catalogue

Failure	What it looks like	Fix
Infinite loop	Calls the same tool over and over with the same arguments	`max_steps`; detect repeated identical calls and break
Cost explosion	A $0.02 task costs $40	Hard budget cap per task; alert on outliers
Wrong tool	Uses `search` when it should use `get_order`	Better descriptions; fewer, more distinct tools
Hallucinated arguments	Invents an order ID that looks plausible	Schema validation + existence check in your code
Gives up too early	"I couldn't find that" when the data exists	Better tool errors; explicit retry guidance in the system prompt
Doesn't stop	Keeps "improving" a finished answer	Clear completion criteria in the prompt; step cap
Context overflow	Fails at step 12 of a long task	Truncate tool results; summarise (Chapter 53)
Context poisoning	An early wrong fact contaminates everything after	Structured state; verify facts before storing them
Silent wrong answer	Confidently wrong, no error anywhere	Evals (Chapter 49) and groundedness checks
Prompt injection	Content it reads changes its behaviour	Chapter 59–61 — the whole of Part 14

Observability: what you must log

You cannot debug what you cannot see. Log every step of every run:

{
  "trace_id": "abc-123",            # one ID across the whole task
  "step": 3,
  "timestamp": "...",
  "messages_tokens": 4210,          # watch this grow
  "model": "...",
  "tool_called": "get_order_status",
  "tool_args": {"order_id": "ORD-9931"},
  "tool_result_summary": "status=shipped",   # or a hash if sensitive
  "tool_duration_ms": 82,
  "cost_usd": 0.0041,
  "cumulative_cost_usd": 0.0138
}

Then you can answer the questions that actually come up: Why did this cost so much? Where did it go wrong? Which tool is slow? Which tool is never used?

This is the same lesson as monitoring in general (Chapter 49): the trace is the product. Frameworks call this "tracing" or "observability"; LangSmith, Langfuse, Braintrust, Phoenix, and OpenTelemetry's GenAI conventions all do it.

Testing an agent

Unit tests don't work (non-deterministic). What does:

   1. TOOL TESTS (deterministic!)
      Test each tool function normally, with unit tests. Most agent bugs
      are actually tool bugs.

   2. TRAJECTORY TESTS
      For a fixed input, assert on the SEQUENCE of tools called --
      not the exact wording of the answer.
        assert "get_order_status" in [c.name for c in trace.calls]
        assert "issue_refund" not in [...]      # <-- negative assertions matter

   3. END-TO-END EVAL SET (Chapter 49)
      20-200 real tasks, with checks:
        * did it reach the right outcome?
        * did it stay under N steps and $X?
        * did it avoid forbidden tools?
        * LLM-judge for open-ended quality

   4. ADVERSARIAL SET
      Inputs designed to make it misbehave (Chapter 59).
      Run these in CI. Every time.

Negative assertions are the important ones. "Did not call issue_refund" catches more real bugs than "produced a nice answer".

Practice (30 min)

A. Add tracing to your Chapter 51 agent — print the dict above at every step. Run five different tasks. Which was most expensive, and why?

B. Cause each of the first four failures deliberately:

A tool that always returns {"error": "try again"} → watch it loop. Now add repeated-call detection.
Remove max_steps and give it an impossible task. Watch the cost. (Set a low budget cap first!)
Two tools with near-identical descriptions → watch it pick wrong.
Ask about a non-existent record → does it invent one?

C. Write five trajectory tests for your agent, including at least two negative assertions.

Check yourself

Name four ways agents fail.
Why don't unit tests work for the agent as a whole?
What is a trajectory test, and why is a negative assertion valuable?
What must you log at every step?

End of Part 12 — Milestone check

I can explain the difference between a workflow and an agent
I can name the five workflow patterns
I have built a working agent loop from scratch
I know the five safety limits every loop needs
I can write a good tool description and explain why it's a prompt
I know why identity must never be a model argument
I can explain the four layers of agent memory
I know why 5 agents at 90% gives 59%