Chapter 46 — Saving tokens and money

The levers, ordered by impact

1. Use a smaller model when you can. Route by difficulty: a cheap model handles routine requests; escalate only when needed. Typically 5–20x savings.

def route(question):
    if len(question) < 100 and not needs_reasoning(question):
        return "small-cheap-model"
    return "large-capable-model"

2. Make your prompt prefix-stable (prompt caching). Providers cache the computed KV for a repeated prompt prefix, charging much less and responding much faster.

GOOD  (cacheable prefix):                BAD (busts the cache every time):
  [system prompt]        <- stable         [today's date and time]   <- changes!
  [tool definitions]     <- stable         [system prompt]
  [few-shot examples]    <- stable         [user question]
  [retrieved context]                      [tool definitions]
  [user question]        <- varies

One changed token near the front invalidates the whole cached prefix. Put volatile things (timestamps, request IDs, the user's message) at the end.

3. Retrieve instead of stuffing. The 13x saving from Chapter 39. This is Chapter 47.

4. Control output length. Output tokens cost the most and take the most time. Set max_tokens. Ask for brevity. Use structured output so the model can't ramble.

5. Cache responses. Exact-match caching on (model, params, prompt) is free and instant for repeats. Semantic caching (return a cached answer if a new question is very similar) catches paraphrases — but set the similarity threshold carefully, or you'll serve subtly wrong answers.

6. Batch independent items.

Instead of 20 calls, each with a 500-token system prompt:
   -> 1 call classifying 20 items, one 500-token system prompt total

7. Compress context. Summarise long histories; drop low-scoring chunks; strip boilerplate, navigation, and repeated headers from documents before indexing.

Instrument everything

Log, per request: model, input tokens, cached tokens, output tokens, latency, cost, and which feature triggered it. You cannot optimise what you don't measure, and token usage creeps up silently as prompts evolve.

def log_call(response, feature, prices):
    u = response.usage
    cost = (u.prompt_tokens/1e6)*prices["in"] + (u.completion_tokens/1e6)*prices["out"]
    logger.info({"feature": feature, "in": u.prompt_tokens,
                 "out": u.completion_tokens, "cost_usd": round(cost, 6)})

Add a CI check that asserts your key prompt templates haven't grown beyond a token threshold.