Chapter 33 — How an LLM is trained (pretraining)

The training objective — you already know it

Recall from Chapter 10: cross-entropy loss = −log(probability assigned to the correct answer). For an LLM, the "correct answer" is simply the word that actually came next in the text.

Training text:  "The cat sat on the mat"

The model gets 5 training examples from this ONE sentence, all at once:

   given "The"                    predict "cat"
   given "The cat"                predict "sat"
   given "The cat sat"            predict "on"
   given "The cat sat on"         predict "the"
   given "The cat sat on the"     predict "mat"

For each, compute -log(probability the model gave the true word). Average them.
That's the loss. Backpropagate. Update the weights. Next batch.

No human labelled these next-token targets by hand. The text labels itself — that's why it's called self-supervised, and it's why training on internet scale text is possible. (This is the payoff of the concept introduced in Chapter 2.)

Thanks to the causal mask (Chapter 24), all five predictions happen in one forward pass, in parallel. That's why Transformers train so fast.

What the data looks like

A modern model is trained on something like 10–20 trillion tokens, roughly:

Filtered web pages (Common Crawl, cleaned)        ~60-85%
Code (GitHub, permissively licensed)               ~5-20%
Books, academic papers, Wikipedia                  ~5-15%
Q&A and forums (StackExchange etc.)                 few %
Maths (problems, proofs)                            few %
Other languages                                     varies
Synthetic (model-generated textbooks, rewrites)    growing

The cleaning pipeline matters as much as the architecture:

raw HTML
  -> extract the actual text, drop navigation/ads/boilerplate
  -> detect language, filter
  -> quality filters (too short? too many symbols? machine-generated spam?)
  -> remove toxic content and personal data
  -> DEDUPLICATE (exact and near-duplicate)        <- huge quality win
  -> DECONTAMINATE (remove anything matching known test sets)
  -> tokenize -> pack into fixed-length sequences -> shuffle

Deduplication is one of the highest-value steps: duplicated text causes memorisation instead of generalisation.

Scaling laws: the reason the field moved so fast

Researchers discovered that model quality improves as a smooth, predictable power law as you increase parameters, data, and compute — over many orders of magnitude, with no plateau in sight.

This turned AI research from guesswork into planning: you can predict the loss of a model before training it.

The two landmark results:

• Kaplan et al. (2020): given a compute budget, mostly spend it on a bigger model. GPT-3 followed this: 175 billion parameters trained on only 300 billion tokens.
• Chinchilla / Hoffmann et al. (2022): corrected the analysis. Scale model size and data together — roughly 20 tokens per parameter. They trained a 70B model on 1.4 trillion tokens, and it beat a 280B model trained with the same total compute.

That result reset the field: models got smaller and datasets got much bigger.

The twist you should know: Chinchilla optimises training cost. But you train a model once and serve it billions of times. So real products deliberately "over-train" small models — Llama-3-8B was trained on ~15 trillion tokens, about 1,900 tokens per parameter, roughly 95x past Chinchilla-optimal. It costs more to train and far less to run forever. This is why 8-billion-parameter models keep getting dramatically better without getting bigger.

The scale, made concrete

Rough compute for training:   6 x (parameters) x (tokens)  operations

A 7B model on 1 trillion tokens:
   6 x 7e9 x 1e12  =  4.2 x 10^22 operations

On ~1,000 modern GPUs running efficiently: a few weeks.
Cost: roughly $1-10 million in compute, plus data work, plus failed runs.
Frontier models: 10-100x that.

This is why very few organisations pretrain frontier models — and why you should almost always fine-tune an existing open model instead (Chapter 42).

What the model has after pretraining

A base model: a superb text-continuation engine, and not an assistant.

You:  "What is the capital of France?"

Base model output:
  "What is the capital of Germany? What is the capital of Italy?
   What is the capital of Spain? Answer key: 1. Paris 2. Berlin..."

It's not being unhelpful — it's doing its job perfectly. On the internet, a line like that is usually followed by more quiz questions, not an answer. Making it behave like an assistant is a separate step: Chapter 34.

# Compute a real model's loss (cross-entropy) on held-out text.
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch, math

name = "Qwen/Qwen2.5-0.5B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name); model.eval()

text = open("input.txt", encoding="utf-8").read()[:20000]   # Shakespeare
ids = tok(text, return_tensors="pt").input_ids[:, :1024]

with torch.no_grad():
    out = model(ids, labels=ids)
print("cross-entropy loss:", round(out.loss.item(), 3))
print("perplexity:", round(math.exp(out.loss.item()), 1))