Chapter 29 — Tokens in practice: cost, context, and fairness

This chapter is the practical payoff. It affects your bills and your users.

Rule of thumb for English

1 token  ~=  4 characters  ~=  0.75 words
100 tokens ~= 75 words     ~= a short paragraph
1,000 tokens ~= 750 words  ~= 1.5 pages

Your API bill, your rate limits, and your context window are all measured in tokens, not words or characters.

Where tokenization goes wrong

1. Numbers and arithmetic.

Older tokenizers merge frequent digit strings, so numbers split inconsistently:

"2023"     -> 1 token   (common year, got merged)
"2024"     -> maybe 2-3 tokens
"1234567"  -> splits unpredictably, e.g. ["123", "45", "67"]

The model sees inconsistent chunks, which is a genuine cause of arithmetic errors — it's trying to do maths on fragments that don't align to place value. Newer models (Llama 3, GPT-4o's o200k) split digits more consistently, which helps. Practical advice: for real arithmetic, give the model a calculator tool (Chapter 48), don't trust mental maths.

2. Whitespace and code.

"the" and " the" (with a leading space) are different tokens. Runs of spaces used for code indentation may or may not be single tokens. Identifiers fragment:

"getUserById"  ->  ["get", "User", "By", "Id"]     4 tokens

This is why code models often have specialised tokenizers with dedicated whitespace tokens.

3. Language inequality — this one matters ethically.

Tokenizers are trained mostly on English text, so English gets efficient tokens and everyone else pays more:

Consequences: non-English users pay more per sentence, fit less text in the same context window, and often get lower quality (rarer tokens are less well trained). Newer multilingual tokenizers (larger vocabularies) narrow this gap but don't close it.

4. Glitch tokens.

Some tokens exist in the vocabulary because they appeared in the tokenizer's training data (scraped usernames, boilerplate), but were almost never seen in context while the model itself trained. Their embeddings are essentially random, and prompting a model with them can cause bizarre behaviour. The famous example is SolidGoldMagikarp. Modern models are increasingly cleaned of these.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

samples = {
    "English": "The weather is nice today and I plan to go for a walk.",
    "Spanish": "El clima esta agradable hoy y planeo salir a caminar.",
    "German":  "Das Wetter ist heute schoen und ich plane spazieren zu gehen.",
    "French":  "Le temps est agreable aujourd'hui et je compte aller marcher.",
}

print(f"{'language':10} {'chars':>6} {'words':>6} {'tokens':>7} {'tok/word':>9}")
for lang, text in samples.items():
    n = len(enc.encode(text))
    w = len(text.split())
    print(f"{lang:10} {len(text):6d} {w:6d} {n:7d} {n/w:9.2f}")

Estimating cost before you call an API

import tiktoken

def estimate(prompt, expected_output_tokens, in_price, out_price, model="gpt-4o-mini"):
    """prices are dollars per 1 million tokens"""
    enc = tiktoken.encoding_for_model(model)
    n_in = len(enc.encode(prompt))
    cost = (n_in / 1e6) * in_price + (expected_output_tokens / 1e6) * out_price
    return n_in, cost

prompt = "Summarise the following document:\n\n" + ("lorem ipsum " * 2000)
n, cost = estimate(prompt, expected_output_tokens=300,
                   in_price=0.15, out_price=0.60)     # illustrative prices
print(f"{n} input tokens, estimated ${cost:.4f} per call")
print(f"At 100,000 calls/day: ${cost*100000:.2f}/day")

Note: prices change constantly — always check the provider's current pricing page. The method is what matters: count tokens, multiply, multiply again by your call volume, before you build.