Chapter 77 — Deploying agents to production

A real-world reference point

In February 2024, Klarna announced that its AI assistant was handling about two-thirds of its customer-service chats in its first month, the work of roughly 700 human agents. In May 2025, its CEO said the company was recruiting human agents again, because cost had been "a too predominant evaluation factor" and the result was lower quality. Separately, in February 2024 a Canadian tribunal ruled (Moffatt v. Air Canada) that Air Canada was liable for what its website chatbot told a customer about bereavement fares, even though the bot was wrong.

Neither story is about model quality alone. They are about operations: measuring outcomes in production (Chapter 76), escalating to humans, constraining what the agent can promise, and owning the consequences. That's what "deploying an agent" really means.

The reference architecture

                 users / webhooks / cron / queues
                               |
                     +---------v---------+
                     |   API / GATEWAY   |  auth, per-tenant quotas,
                     +---------+---------+  input guardrails (Ch 60)
                               |  enqueue task (run_id)
                     +---------v---------+
                     |    TASK QUEUE     |  backpressure, priorities,
                     +---------+---------+  retries with backoff
                               |
          +--------------------v---------------------+
          |   AGENT WORKERS (stateless processes)    |
          |   harness + loop (Ch 74), executed via a |
          |   DURABLE RUNTIME: every model call and  |
          |   tool call is a journaled step          |
          +----+-------------+-------------+---------+
               |             |             |
     +---------v--+   +------v-------+  +--v---------------------+
     | LLM GATEWAY|   | TOOL GATEWAY |  | SANDBOXES              |
     | routing,   |   | MCP servers, |  | container / microVM    |
     | fallbacks, |   | authz per    |  | per task; no secrets;  |
     | caching,   |   | user, idem-  |  | egress allowlist;      |
     | budgets,   |   | potency keys,|  | destroyed afterwards   |
     | pinned     |   | rate limits  |  |                        |
     | versions   |   |              |  |                        |
     +------------+   +--------------+  +------------------------+
               \             |             /
                +------------v------------+
                |  STATE + OBSERVABILITY  |  run journal / checkpoints,
                |                         |  traces (OpenTelemetry GenAI),
                |                         |  cost + eval dashboards, alerts
                +-------------------------+

Let's walk through the parts that differ from ordinary web services.

1. Durable execution: surviving crashes, deploys, and long waits

Your worker will die mid-task: a deploy, an out-of-memory kill, a spot instance reclaimed. If the agent's state lives in a Python list, the task is lost, or worse, restarted from scratch and the refund issued twice.

Durable execution fixes this by journaling every step. Each model call and tool call gets a deterministic step key and its result is written to durable storage before the agent moves on. To resume after a crash, you simply run the same code again: completed steps return their recorded results instantly (replay), and execution continues from the first unfinished step. Workflow engines such as Temporal, Restate, DBOS, and cloud step-function services do this, as do agent frameworks' checkpointers (LangGraph, Chapter 69). The core fits in about 40 lines, plus a demo:

"""durable.py -- a journal that makes an agent run survive crashes and redeploys.

Every model call and tool call is a STEP with a deterministic key
(run_id, step_no). Before doing a step, look it up: if it already finished,
return the recorded result instead of doing it again. A crashed run is
resumed by simply calling the same code again -- finished steps replay from
the journal in milliseconds, and execution continues from the first
unfinished one. This is the core idea behind Temporal, Restate, DBOS, and
LangGraph's checkpointers, in ~40 lines.
"""
import json
import sqlite3


class Journal:
    def __init__(self, path="runs.db"):
        self.db = sqlite3.connect(path)
        self.db.execute("""CREATE TABLE IF NOT EXISTS steps(
            run_id TEXT, step_no INTEGER, kind TEXT, result TEXT,
            PRIMARY KEY (run_id, step_no))""")

    def step(self, run_id, step_no, kind, fn):
        row = self.db.execute(
            "SELECT result FROM steps WHERE run_id=? AND step_no=?",
            (run_id, step_no)).fetchone()
        if row:                                   # already done: REPLAY
            return json.loads(row[0])
        result = fn()                             # not done: DO IT
        self.db.execute("INSERT INTO steps VALUES (?,?,?,?)",
                        (run_id, step_no, kind, json.dumps(result)))
        self.db.commit()                          # durable BEFORE we move on
        return result


# ---- the side-effecting tool must ALSO be idempotent ------------------------
# The crash can land between "refund API succeeded" and "journal committed".
# On resume the step re-runs, so the refund API must recognise the retry.
def issue_refund(order_id, amount, idempotency_key, payments):
    if idempotency_key in payments:              # the payment provider's job;
        return payments[idempotency_key]         # Stripe, Adyen etc. support this
    receipt = {"refund_id": f"R-{len(payments) + 1}", "order_id": order_id,
               "amount": amount}
    payments[idempotency_key] = receipt
    return receipt


if __name__ == "__main__":
    import os
    if os.path.exists("/tmp/runs_demo.db"):
        os.remove("/tmp/runs_demo.db")
    payments = {}                                 # stands in for the payment provider
    calls = {"model": 0}

    def fake_model(step):
        calls["model"] += 1
        return {"tool": "issue_refund", "args": {"order_id": "ORD-9931", "amount": 25}} \
            if step == 0 else {"answer": "Refunded $25 to ORD-9931."}

    def run(run_id, crash_after_refund=False):
        j = Journal("/tmp/runs_demo.db")
        step = 0
        while True:
            decision = j.step(run_id, step, "model", lambda: fake_model(step))
            step += 1
            if "answer" in decision:
                return decision["answer"]
            key = f"{run_id}:{step}"              # deterministic idempotency key
            j.step(run_id, step, "tool", lambda: issue_refund(
                **decision["args"], idempotency_key=key, payments=payments))
            step += 1
            if crash_after_refund:
                raise SystemExit("power cut!")

    try:
        run("run-42", crash_after_refund=True)
    except SystemExit as e:
        print("first attempt:", e)
    print("resumed:", run("run-42"))
    print("refunds issued:", len(payments), "| model calls:", calls["model"])
    assert len(payments) == 1 and calls["model"] == 2

Output:

first attempt: power cut!
resumed: Refunded $25 to ORD-9931.
refunds issued: 1 | model calls: 2

The crash happened right after the refund. On resume, the model call and the refund were replayed from the journal, not repeated: one refund, and no paying for the same model call twice. Durable execution also makes human-in-the-loop waits cheap: the run simply stops at "waiting for approval" with its state in the journal, holds no process or memory for three days, and resumes when the approval event arrives.

2. Idempotent tools: the other half of "exactly once"

Look again at issue_refund above. Even with a journal, there is a tiny window: the payment API succeeded, then the process died before the journal write. On resume, the step re-runs. The only defence is an idempotency key: a deterministic ID (run_id:step) sent with the request, so the payment provider recognises the retry and returns the original result instead of paying twice. Major payment APIs support this directly; build the same into your own side-effecting tools.

Rule: every tool that changes the world takes an idempotency key, and the key is derived from the run and step, never randomly generated per attempt.

3. Sandboxing: one disposable box per task

Agents that run code or shell commands need isolation stronger than "the same container as the web server":

Per-task rules (Chapter 61, made concrete): a fresh sandbox per task, destroyed afterwards; no production credentials inside, with tools that need secrets called through the tool gateway, which holds the credentials and checks the user's permissions; an egress allowlist (package registry, your git host, nothing else); and CPU, memory, disk, and wall-clock limits.

4. The LLM gateway: routing, fallbacks, caching, budgets

Don't let every worker call model providers directly. A gateway (your own code or an off-the-shelf one) gives you one place for:

• Version pinning. Use dated model IDs, not floating aliases, in production. A silent model update can change behaviour as much as a prompt change, so treat a model upgrade like a deploy: eval first (Chapter 76).
• Fallbacks. Provider outage or rate limit → retry with backoff → fall back to a secondary model that has passed your evals.
• Routing. Send easy steps to a small, cheap model and hard ones to a frontier model (Chapter 46). Chapter 80 shows routing with a dedicated decision model.
• Prompt caching. Stable prefixes (system prompt, tools, rules) are billed at a steep discount when cached. For agents that re-send a long prefix every step, this is often the single biggest cost lever.
• Budgets. Per-task, per-user, per-tenant spend limits enforced outside the agent, plus anomaly alerts ("this task has made 400 calls").

5. Observability: traces as the product

Chapter 55's per-step log becomes a distributed trace: one trace per run, with nested spans for each model call, tool call, and sub-agent. The OpenTelemetry GenAI semantic conventions standardise the names, so any backend (Langfuse, Phoenix, Datadog, Honeycomb, and others) can display them:

   trace run-42  (invoke_agent support-agent)            8.4 s   $0.031
   |- chat claude-...           gen_ai.usage.input_tokens=3120  1.9 s
   |- execute_tool verify_identity                              0.2 s
   |- chat claude-...           (cached prefix: 2,900 tokens)   1.1 s
   |- execute_tool issue_refund  idempotency_key=run-42:3       0.6 s
   |- chat claude-...                                           1.4 s

Attributes such as gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens are part of the convention (still marked experimental, so check the current version). Add your own: tenant, run_id, prompt_version, harness_version, tools_denied, cost_usd.

The dashboards that matter for agents:

6. Rollouts: prompts and models are deploys

Treat prompt, tool, harness, and model changes as code deploys: versioned, reviewed, evaluated, and rolled out gradually.

1. Offline eval gate (Chapter 76): the change must not regress pass^k, safety rules, or cost.
2. Shadow mode: run the new version on real inputs without acting (tools are dry-run), and compare decisions with the current version.
3. Canary: send 1–5% of new tasks to the new version; watch the dashboard above; widen gradually.
4. Instant rollback: the version is a config value, not a rebuild.

There is one agent-specific wrinkle. Long-running tasks span deploys. A task that started on version 12 shouldn't wake up halfway through on version 13 with a different prompt and tool set. Anthropic described using rainbow deployments for its research agents: old and new versions run side by side, new tasks go to the new version, and in-flight tasks finish on the version they started on. Durable runtimes support the same idea with workflow versioning.

7. Rate limits and backpressure

Ten thousand agents each making a model call every two seconds will hit provider rate limits. Use a queue with concurrency limits per provider and per tenant, exponential backoff with jitter on 429/529 errors, priorities (interactive users before overnight batch jobs), and batch APIs for non-urgent work (typically about half price).

8. Incidents: the agent runbook

Write this before launch:

• Kill switch: one flag that stops all tool execution (Chapter 61), and one per tool. Practise using it.
• Blast-radius limits: max refunds per hour, max emails per run, max value per action, enforced in the tool gateway.
• Replay: from a trace, re-run a failing task in a sandbox with the same inputs to reproduce it.
• Customer remediation: who contacts the customer when the agent got it wrong, and who can reverse its actions.
• Post-incident: the failing conversation becomes an eval task (Chapter 76).

Hosted agent platforms vs building it yourself

Several providers now offer managed agent runtimes: you supply the prompt, tools, and configuration, and they run the loop, sandboxes, state, and scaling for you. Cloud vendors offer agent hosting services too.

Either way, the evals, budgets, idempotent tools, and runbook are your job. No platform knows your refund policy.

The production checklist

   [ ] durable execution: crash mid-task -> resumes, no repeated side effects
   [ ] every side-effecting tool takes a deterministic idempotency key
   [ ] fresh sandbox per task; no prod secrets inside; egress allowlist
   [ ] model versions pinned; fallback model has passed the eval suite
   [ ] per-task / per-user / per-tenant budgets enforced outside the agent
   [ ] OpenTelemetry traces with cost, tokens, tool calls, versions
   [ ] dashboards: success, cost p95, steps p95, tool errors, escalations
   [ ] offline eval gate in CI for prompt / tool / harness / model changes
   [ ] shadow -> canary -> full rollout; one-click rollback
   [ ] in-flight tasks finish on their starting version
   [ ] kill switch tested; blast-radius limits in the tool gateway
   [ ] human escalation path, and a named owner for the agent's actions