Chapter 53 — Memory: how an agent remembers

In one sentence

An agent has no memory at all, so "memory" means deciding what to put back into the context window on each call — and the whole craft is choosing what to keep.

The problem

From Chapter 31: the model is stateless. Every API call is independent. What feels like memory is your code re-sending the conversation each time.

But conversations grow, and the context window is finite (Chapter 45). A long-running agent will overflow it. So you must decide what to carry forward.

The four layers of memory

   +--------------------------------------------------------------+
   | 1. WORKING CONTEXT   the last few turns, verbatim             |
   |                      -> always included                       |
   +--------------------------------------------------------------+
   | 2. ROLLING SUMMARY   older turns, compressed by the LLM       |
   |                      -> "so far, the user wants X, we found Y"|
   +--------------------------------------------------------------+
   | 3. RETRIEVED MEMORY  older facts, fetched only when relevant   |
   |                      -> RAG (Chapter 47) over past messages    |
   +--------------------------------------------------------------+
   | 4. STRUCTURED STATE  exact facts, in a real data structure     |
   |                      -> {"customer_id": "C-42",                |
   |                          "order": "ORD-9931", "step": 3}       |
   +--------------------------------------------------------------+

Layer 4 is the one beginners skip and shouldn't. Anything that must be exact — an ID, a date, a running total, which step you're on — should live in a Python dict, not in prose. Prose summaries drift and hallucinate; a variable doesn't.

   BAD:  summary = "The user is asking about their order, I think it was 9931"
   GOOD: state = {"customer_id": "C-42", "order_id": "ORD-9931", "step": "verify"}
         # then render it into the prompt deterministically

The pattern that works

def build_context(state, history, user_message, max_tokens=8000):
    parts = [
        SYSTEM_PROMPT,                      # stable -> enables prompt caching (Ch 46)
        render_state(state),                # exact facts, deterministic
        render_summary(history.summary),    # compressed older turns
        *history.recent_turns[-6:],         # last few turns verbatim
        user_message,                       # variable content LAST
    ]
    while count_tokens(parts) > max_tokens:
        drop_oldest_recent_turn(parts)      # then re-summarise
    return parts

Note the ordering — it's the same rule as Chapter 45/46: stable content first (so prompt caching works), variable content last (so the model attends to it).

Summarise on a schedule, not every turn

   every turn:      keep the last N turns verbatim (cheap)
   every N turns:   ask the LLM to update a running summary of everything older
                    (costs one extra call -- don't do it every message)
   on important
   events:          extract structured facts into state
                    ("the user confirmed order ORD-9931")

Forgetting is a feature

Unbounded memory becomes noise and cost. Deliberately:

Time-to-live on stored memories
Relevance decay — old, unused facts rank lower
Deduplicate — the same fact stated five ways is one fact
Scope it — per-user, per-session, per-tenant. Never let one user's memory leak into another's (this is a real, serious bug class — see Chapter 61).

Practice (30 min)

A. Watch the context grow. Take your Chapter 51 agent and print count_tokens(messages) at every step. Have a five-step conversation. Plot it. That curve is your cost curve.

B. Add a summary layer: after every 6 messages, replace the oldest 4 with an LLM-generated summary. Compare total tokens over a 20-turn conversation, with and without.

C. Add structured state: keep a dict of facts the agent has confirmed (names, IDs, decisions). Render it at the top of every prompt. Notice how much more reliable it is than hoping the model remembers.

D. Break it: run a conversation until it exceeds the context window. What error do you get, and how would you have prevented it?

Common confusions

"Does the model learn from our conversation?" No. Nothing is stored in the weights. Fine-tuning (Chapter 42) changes weights; conversation does not.
"Isn't a bigger context window the answer?" It helps, but it's expensive (Chapter 46) and quality degrades in the middle (Chapter 45). Curation beats capacity.
"Should I use a memory framework?" Try the four layers by hand first — it's fifty lines. Then a framework will make sense rather than being magic.

Check yourself

Why does an agent have no memory of its own?
What are the four layers, and which should hold an order ID?
Why summarise on a schedule rather than every turn?
Why does memory need to be scoped per user?