Part 7 — Tokens

Short part, big practical payoff. Tokens determine what you pay, how much text fits in the model's memory, why it's bad at spelling and arithmetic, and why non-English users get a worse deal.

Chapter 27 — Computers don't see words

In one sentence

Before an LLM sees anything, your text is chopped into "tokens" and each token becomes a number — and the choice of how to chop matters enormously.

The problem

Your model's embedding table has one row per known item. So you must decide: what is an item?

Option	Vocabulary size	Sequence length	Problem
Characters	~100	very long (4–5x more items)	long sequences are expensive (remember: attention is quadratic!); model must relearn spelling
Words	200,000+	short	unknown words break it: typos, names, new slang, numbers, other languages -> `<UNK>`, and information is lost
Subwords	32,000–256,000	moderate	the sweet spot — this is what everyone uses

Subword tokenization keeps common words whole and splits rare ones into meaningful pieces:

"the"            -> ["the"]                    common, one token
"unbelievable"   -> ["un", "believ", "able"]    rarer, three pieces
"Kubernetes"     -> ["Kub", "ernet", "es"]      rare, still representable
"asdkjhasd"      -> ["as", "d", "k", "j", ...]  gibberish, falls back to fragments

Nothing is ever truly unknown, and common text stays compact. This is exactly the FastText idea from Chapter 20 — a word is built from smaller pieces — applied to model inputs.

The vocabulary is fixed forever

The tokenizer is trained once, before the model, and then frozen. Every token ID maps to a row in the embedding matrix. Change the tokenizer and every ID shifts, invalidating the entire model. A model and its tokenizer are married.

Practice (5 min) — see it happen

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")     # used by GPT-3.5/GPT-4

for s in ["Hello world",
          "unbelievable",
          "antidisestablishmentarianism",
          "1234567",
          "def fibonacci(n):",
          "   indented code",
          "Bonjour le monde"]:
    ids = enc.encode(s)
    pieces = [enc.decode([i]) for i in ids]
    print(f"{s!r:35}  {len(ids):2d} tokens  {pieces}")

Look closely at the output. Notice:

Leading spaces are attached to words — " world" is one token, and it's a different token from "world".
Long rare words shatter into several pieces.
Numbers split in ways that look arbitrary.

Also try the browser tool: platform.openai.com/tokenizer — paste any text and see it coloured by token. Do this now; it takes 30 seconds and makes the concept concrete.

Check yourself

Why not just use words as tokens?
Why not use characters?
Why can't you swap a model's tokenizer?