Chapter 48 — Tools, structured output, and agents

Structured output: making the model return parseable data

Asking nicely for JSON works ~90–98% of the time — which is a failure every 20 requests. Three levels of reliability:

Level 1 — Ask and validate.

prompt = 'Extract the fields. Return ONLY JSON: {"name": str, "amount": float}'
# then json.loads() in a try/except, and retry on failure

Level 2 — JSON mode. A provider flag that forces syntactically valid JSON (though not necessarily your schema).

Level 3 — Schema-constrained decoding. At each step, the sampler is only allowed to pick tokens that keep the output valid against your schema. For the supported schema, invalid output becomes impossible at the decoder level, not merely unlikely — assuming the schema and constrained-decoding implementation are correct.

from pydantic import BaseModel

class Expense(BaseModel):
    item: str
    amount_eur: float
    approved: bool
    reason: str

# Supported natively by many providers (structured outputs / response_format)
# and by local servers via libraries like Outlines, XGrammar, or llama.cpp grammars.

Tip: if you need an explanation, ask for a short user-visible field like evidence or rationale_summary, not hidden chain-of-thought. Constrained decoding forces field order, so put fields in the order the answer should be assembled: evidence first, then the final structured values.

Tools (function calling)

Give the model a menu of functions. It decides when to call one and with what arguments; your code executes it and returns the result.

tools = [{
    "type": "function",
    "function": {
        "name": "get_holiday_balance",
        "description": "Get an employee's remaining holiday days for this year.",
        "parameters": {
            "type": "object",
            "properties": {"employee_id": {"type": "string"}},
            "required": ["employee_id"],
        },
    },
}]

# The loop:
#  1. send messages + tools
#  2. if the model returns a tool call -> run YOUR function
#  3. append the result as a "tool" message
#  4. send again -> the model writes the final answer

This is how you fix the model's weaknesses:

The mental model from Part 2 returns: the LLM handles language and decisions; deterministic code handles facts and actions. Neither half is sufficient alone.

Agents: models that loop

An agent is an LLM in a loop with tools and a goal:

   goal
     |
     v
   +-------------------------------------+
   |  1. LLM decides the next action     |
   |  2. your code executes the tool     |
   |  3. result is appended to context   |<---+
   |  4. repeat until done or budget out |    |
   +-------------------------------------+----+
     |
     v
   answer

Agents are powerful and fragile. The safety rules are non-negotiable:

1. Least privilege. Read-only by default. Give the minimum set of tools.
2. Human approval for consequential actions — sending, paying, deleting, deploying. This is the single most effective control.
3. Hard limits. Max iterations, max tokens, max spend, timeouts. Always.
4. Sandbox any code execution. No secrets, no network unless allowlisted, ephemeral filesystem.
5. Validate every tool argument in your code before executing. Never trust model-generated paths, URLs, recipients, or SQL.
6. Log every tool call and alert on anomalies.