26. Agentic AI Systems

"Design an AI agent platform" (or "design a coding agent", "design a customer-support agent that can take actions") is now a common senior-level prompt. It builds on §25 (RAG, LLM serving) but adds what makes agents hard: multi-step autonomy, side effects through tools, long-running execution, and an attack surface where untrusted text can steer behavior.

26.1 Workflows vs agents: the first design decision

Anthropic's engineering guide "Building effective agents" (December 2024) draws a useful line:

• Workflows: LLMs and tools orchestrated through predefined code paths. Patterns: prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer.
• Agents: the LLM dynamically directs its own process and tool use, looping until done.

Rule of thumb: use the least autonomy that solves the problem. A workflow is cheaper, faster, more predictable, and easier to evaluate. Choose an agent only when the steps can't be known in advance (open-ended coding, research, troubleshooting). Saying this out loud in an interview is a strong signal.

26.2 The agent loop

 ┌─────────────────────────────────────────────────────────────────────────┐
 │ task + context (system prompt, user goal, memory, tool definitions)      │
 └───────────────┬─────────────────────────────────────────────────────────┘
                 ▼
          ┌────────────┐   tool_call(name, args)   ┌───────────────────────┐
          │    LLM     │──────────────────────────▶│ Tool gateway           │
          │ (reason +  │                           │ authz · validate args  │
          │  choose)   │◀──────────────────────────│ approval? · execute    │
          └─────┬──────┘   tool_result (untrusted!) │ (sandbox / API)        │
                │                                   └───────────────────────┘
                │ loop until: final answer | step budget | token/cost budget | timeout
                ▼                                     | human escalation
          result + full trajectory (every step logged & traced)

26.3 Platform architecture

 Clients (chat UI, IDE, API, schedulers) ─▶ API gateway (authN, tenant, quotas)
   ─▶ Agent runtime (DURABLE execution: each step persisted; resumable; cancellable)
        ├─ Context manager: system prompt, task, retrieved memory, history compaction
        ├─ Model gateway: routing (big vs small model), fallbacks, prompt caching, budgets
        ├─ Policy engine: which tools, which data, which actions need human approval
        ├─ Tool gateway ──▶ internal APIs (scoped, on-behalf-of user tokens)
        │                ──▶ MCP servers (standard tool/resource protocol)
        │                ──▶ Sandbox pool (code execution: microVMs/gVisor, no ambient creds,
        │                                  egress allowlist, CPU/mem/time limits)
        ├─ Memory: short-term (conversation/scratchpad) · long-term (vector + structured facts,
        │          per-user/tenant isolated, with provenance & expiry)
        └─ Human-in-the-loop service: approval requests, timeouts, audit
   ─▶ Observability: per-step traces (prompt, tool calls, tokens, latency, cost), redaction
   ─▶ Evaluation platform: offline suites, trajectory grading, online sampling, regression gates

26.4 Estimates (calibrating cost and capacity)

1M agent tasks/day; avg 15 steps/task; avg 12k input tokens + 500 output tokens per step
  input tokens:  1M × 15 × 12k = 180B tokens/day
  output tokens: 1M × 15 × 500 = 7.5B tokens/day
Input dominates → PROMPT CACHING of the stable prefix (system prompt + tool defs + early history)
  can cut input cost and time-to-first-token substantially; context compaction keeps per-step
  input from growing linearly with steps
Latency: 15 steps × (model ~2-6 s + tool ~0.2-5 s) → minutes per task → async UX (status, streaming
  progress, notifications), not a synchronous HTTP request
Sandboxes: if 30% of tasks run code, avg 3 min each → 1M × 0.3 × 180 s / 86,400 s ≈ 625 concurrent
  sandboxes avg, ~2-3k at peak → warm pool of pre-booted microVMs to hide boot latency

26.5 Deep dive: durable execution for long-running agents

Agents run for minutes or hours and call flaky tools. Process restarts, deploys, and model timeouts are routine, so the runtime must be durable, the same idea as workflow engines (§12.6):

• Persist each step (model request/response, tool call/result) as an event before acting on it. Resume after a crash from the last completed step.
• LLM outputs are nondeterministic: on replay, use the recorded model output, never call the model again for an already-completed step. (This is the same determinism rule Temporal imposes on workflow code.)
• Tool calls with side effects need idempotency keys (task_id:step_n), so a retry after a crash doesn't send the email twice or create two PRs.
• Budgets enforced by the runtime, not the model: max steps, max tokens, max wall-clock, max cost per task and per tenant. Agents in loops are the new runaway queries.

26.6 Deep dive: tool security and the agent threat model

The core problem is that anything the agent reads can act as instructions: web pages, emails, documents, tool outputs, even file names. Simon Willison's "lethal trifecta" is a useful checklist. An agent that combines (1) access to private data, (2) exposure to untrusted content, and (3) the ability to communicate externally can be tricked into exfiltrating data.

MCP (Model Context Protocol), introduced by Anthropic in November 2024, standardizes how agents discover and call tools and read resources. Platform implications: an MCP gateway that enforces authN/authZ, allowlists servers, scans tool descriptions (tool definitions are themselves untrusted input), and logs calls.

26.7 Deep dive: context and memory

• Context window is a budget. Long tasks overflow it. Strategies: compaction (summarize older steps), retrieval of relevant earlier steps, structured scratchpads (todo lists, notes files), and sub-agents with fresh contexts that return compact results.
• Long-term memory (user preferences, facts learned) needs provenance (which conversation wrote it), expiry, user visibility and deletion (GDPR), and per-user and per-tenant isolation. Memory written from untrusted content is a persistent prompt-injection vector, so treat memory writes as privileged.
• Multi-agent patterns (orchestrator + workers): useful for parallel, separable subtasks (research across many sources). The cost is tokens (each agent re-reads context) and coordination bugs. Use them when parallelism or context isolation pays for itself.

26.8 Deep dive: evaluating agents

Public benchmarks such as SWE-bench (resolving real GitHub issues, verified by tests) show the pattern: evaluate agents in executable environments with objective checks, not only by judging text.

26.9 Failure modes & corner cases

• Infinite or circular loops (retrying the same failing tool): step budgets, loop detection (same call plus same args repeated), and escalation.
• Partial completion: the agent finished 3 of 5 steps, then failed. Expose state clearly, make steps resumable, and give compensating actions for multi-step side effects (saga thinking, §12).
• Tool schema drift: an API changed and the agent sends wrong arguments. Validate arguments against schemas at the gateway and return structured errors the model can correct.
• Secrets in context: an API key in a tool result ends up in logs, memory, or another user's context. Redact at the gateway, and never return secrets to the model.
• Concurrent agents editing the same resource (two agents changing one repository or record): locks or optimistic concurrency at the tool layer.
• Approval timeouts: a human doesn't respond for 3 days. Define expiry and safe defaults (cancel, not proceed).
• Upstream rate limits: 1,000 agents hammer an internal API. Per-tool quotas and backpressure in the tool gateway.
• Cost runaway from one tenant: per-tenant budgets with hard stops and alerts.
• Model or provider outage: fall back to another model with an eval-verified prompt variant. Agents tuned to one model often degrade on another, so evaluate the fallback too.

26.10 Signals by level on this topic