Chapter 39 — What it costs, and how to think about it

The three numbers that determine everything

1. MODEL SIZE      -> how much memory to read per token -> speed and $ per token
2. PROMPT LENGTH   -> prefill cost -> time to first token, and input $
3. OUTPUT LENGTH   -> number of decode steps -> most of the latency, and output $

Pricing shape (why output costs more)

Providers charge per token, and output tokens typically cost 2–5x input tokens. Why? Input tokens are processed in one efficient parallel pass; output tokens each require a separate full pass through the model.

Also increasingly common: cached input tokens at a large discount (often 10–50% of the normal price), when your prompt shares a prefix with a recent one. This is the prefix caching from Chapter 38, exposed as a pricing tier — and it rewards putting your stable content first and your variable content last.

Prices change constantly and vary by model. Always check the provider's current pricing page. What follows is the method, with illustrative numbers.

A worked cost model

A support assistant, 100,000 requests per day:

Per request:
   system prompt + instructions    500 tokens   (same every time -> CACHED)
   retrieved document chunks     2,400 tokens   (varies -> not cached)
   user question + history         600 tokens
   ----------------------------------------
   input total                   3,500 tokens
   output                          350 tokens

Illustrative prices: $0.15 per 1M input, $0.075 per 1M cached, $0.60 per 1M output

   uncached input:  3,000 / 1e6 x 0.15   = $0.00045
   cached input:      500 / 1e6 x 0.075  = $0.0000375
   output:            350 / 1e6 x 0.60   = $0.00021
   -----------------------------------------------------
   per request                            ~ $0.0007
   per day (100k)                         ~ $70
   per month                              ~ $2,100

Now the naive version — skip retrieval, just paste the whole 60,000-token handbook into every prompt:

   input:  60,000 / 1e6 x 0.15 = $0.009
   output:    350 / 1e6 x 0.60 = $0.00021
   ------------------------------------------
   per request  ~ $0.0092     ->  ~$920/day  ->  ~$27,600/month

Same product. ~13x the cost. And the retrieval version usually gives better answers, because the model isn't distracted by 58,000 irrelevant tokens (Chapter 45).

Build this spreadsheet before you build the product. It takes 20 minutes and it will change your design.

The levers, in order of impact

1. Use a smaller model where you can. Route easy requests to a cheap model, escalate hard ones. Often 5–20x savings.
2. Send less context. Retrieve the relevant 3,000 tokens instead of pasting 60,000 (Chapter 47).
3. Exploit prefix caching. Put stable content (system prompt, tool definitions, few-shot examples) first; put the user's variable input last. Never change a token near the front unnecessarily.
4. Cap and shape the output. Set max_tokens. Ask for concise answers. Use structured output so it can't ramble. Output tokens are the expensive ones.
5. Cache responses. Identical or near-identical questions are extremely common in production.
6. Batch independent items into one call where the task allows.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

def cost_model(system_tokens, context_tokens, question_tokens, output_tokens,
               price_in, price_cached, price_out, requests_per_day):
    uncached = context_tokens + question_tokens
    per_req = (uncached/1e6)*price_in + (system_tokens/1e6)*price_cached \
              + (output_tokens/1e6)*price_out
    return per_req, per_req*requests_per_day, per_req*requests_per_day*30

# your own numbers here
for name, ctx in [("RAG (6 chunks)", 2400), ("whole handbook", 60000)]:
    r, d, m = cost_model(500, ctx, 600, 350, 0.15, 0.075, 0.60, 100_000)
    print(f"{name:20} ${r:.5f}/req   ${d:8.2f}/day   ${m:9.2f}/month")