Chapter 45 — The context window: your working-memory budget

What it is

The context window is the maximum number of tokens the model can consider at once — prompt + output together. Everything the model "knows" about your specific situation must fit inside it.

+------------------------------------------------------------------+
|                    CONTEXT WINDOW (e.g. 128,000 tokens)          |
+------------------------------------------------------------------+
| system  | conversation | retrieved  | your     |  room reserved   |
| prompt  |   history    | documents  | question |  for the ANSWER  |
+------------------------------------------------------------------+
                                                  ^
                    reserve this FIRST -- if you fill the window
                    with input, there's no room to reply

Why it's limited (three reasons)

1. Attention is quadratic (Chapter 25). 2x the tokens = ~4x the attention work. Prefill on a 100k prompt is genuinely expensive.
2. The KV cache grows linearly (Chapter 38) with context and with the number of concurrent users. This is usually the binding limit for a serving system.
3. The model was trained on a certain length. Beyond that, positional information is out-of-distribution. (Techniques like RoPE scaling stretch it after training — which is how "128k context" models are made — but quality degrades gradually.)

The most important practical fact: "lost in the middle"

Research (Liu et al., 2023, Lost in the Middle) found that models retrieve information best from the beginning and end of a long context, and worst from the middle — a U-shaped accuracy curve. It persists to varying degrees in current long-context models.

   accuracy
      ^
  high|  *                                       *
      |    *                                  *
      |       *                            *
      |          *                      *
   low|              *  *  *  *  *  *
      +-------------------------------------------> position of the needed
        start            middle             end      information in the prompt

Therefore:

• Put the most important material at the start and the end.
• Restate the question after the documents.
• Rank retrieved chunks and place the best ones at the top and bottom of the context block; bury the weak ones in the middle.
• Fewer, better tokens beat more, mediocre tokens — a tight 6,000-token prompt often beats a sloppy 60,000-token one and costs 10x less.

"Advertised context" is not "usable context." A model that accepts 128k tokens may reliably use far less for hard multi-step tasks. Test it on your own task rather than trusting the headline number.

Budgeting a prompt

total window                       128,000
  - reserve for the answer          -2,000      <-- subtract FIRST
  - system prompt                     -400
  - tool definitions                  -600
  - safety margin (~3%)             -3,800
  ------------------------------------------
  = working budget                 121,200      for history + context + examples

Then decide deliberately: how many retrieved chunks? how many few-shot examples? how much history? A concrete allocation beats "stuff it until it errors".

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

def budget(system, tools, history, chunks, question,
           window=128000, reserve_output=2000, margin=0.03):
    n = lambda s: len(enc.encode(s))
    used = n(system) + n(tools) + n(history) + sum(map(n, chunks)) + n(question)
    limit = int(window * (1 - margin)) - reserve_output
    print(f"used {used:,} / {limit:,} tokens  ({used/limit:.0%})")
    if used > limit:
        print("  -> over budget. Drop the lowest-ranked chunks first.")
    return used <= limit