Part 3 — Era 2: Learning from examples (1990s–2000s)
This part contains the single most important chapter in the guide for understanding LLMs: Chapter 12, on n-gram models. An LLM is a direct descendant of what you'll build there.
Chapter 8 — The big switch: from rules to examples
In one sentence
Stop writing the rules; collect examples of the right answer and let an optimisation procedure find the rules for you.
The shift in three columns
| Rule era | Statistics era | |
|---|---|---|
| Human provides | the knowledge | the examples and the features |
| Machine provides | application of rules | the weights (how much each feature matters) |
| Answers are | certain (true/false) | probable ("87% spam") |
| Unknown input | fails | degrades gracefully |
| Adding knowledge | write another rule | collect more data |
| Explaining a decision | show the rule chain | show which features had the biggest weights |
Why this became possible in the 1990s
Three things arrived at once: digital text (email, the web, digitised corpora), cheap storage, and enough CPU to run optimisation over thousands of examples. Statistical methods had existed for decades; the data hadn't.
The recipe you'll follow for the rest of Part 3
1. Get examples with answers (data + labels)
2. Turn each example into numbers (features)
3. Pick a model shape with knobs (e.g. weighted sum -> probability)
4. Pick a loss (how wrong is a prediction?)
5. Turn the knobs to reduce loss (training)
6. Test on examples it never saw (evaluation)
Memorise these six steps. Every model in this guide — including a $100M LLM — follows exactly them.
Check yourself
- What does the human still provide in the statistics era?
- Why are probabilistic answers an advantage, not a weakness?
Further reading
- Course: Google ML Crash Course — "Descending into ML".
- Video: StatQuest, "Machine Learning Fundamentals: The Confusion Matrix" (you'll need it in Chapter 13).