Chapter 35 — Why LLMs make things up
In one sentence
The model was trained to produce plausible text, not true text — and it has no built-in way to tell the difference.
The mechanism, in five parts
1. The objective rewards plausibility, not truth. Cross-entropy asks: "does this look like the training text?" A fluent, wrong citation looks exactly as much like real text as a fluent, correct one. The loss function cannot tell them apart.
2. There is no "I don't know" by default. The softmax always produces a distribution over every token. There's no special outcome for "insufficient information". Saying "I'm not sure" is just another string the model must have been trained to produce in the right situations — which post-training does, imperfectly.
3. Under-determined prompts get pattern-filled. Ask for a citation the model doesn't have. It knows the shape of citations (author, year, plausible title, DOI format) extremely well. So it generates a perfectly-formatted, entirely fictional one. It's doing pattern completion, which is exactly what it was trained for.
4. Long-tail facts are stored weakly or not at all. "Capital of France" appeared millions of times in training. "The 2019 revenue of a mid-size German logistics firm" appeared maybe never. The model interpolates from similar-looking text.
5. Commitment cascade. Once the model has generated a wrong first token, everything after is conditioned on it — and the model is trained to produce coherent text, so it keeps elaborating consistently on the mistake.
The classic demonstration
Prompt: "Give me three papers by [some researcher] on [some topic],
with DOIs."
Typical output without tools:
- Real-sounding titles that don't exist
- Plausible co-authors who never collaborated
- Perfectly-formatted DOIs that resolve to nothing
Wire the same model to a scholarly search API and the problem largely disappears. That single pattern — offload facts to a system of record, keep the LLM for language — is the most important practical lesson in this guide.
The fixes (in rough order of effectiveness)
| Fix | What it does | Chapter |
|---|---|---|
| Give it the source text (RAG) | it answers from documents you provide, not memory | 47 |
| Give it tools | calculator, search, database, code execution — exact systems for exact tasks | 48 |
| Instruct it to ground | "Answer only using the context below. Cite the source. If it isn't there, say you don't know." | 44 |
| Lower the temperature | less randomness on factual tasks | 37 |
| Ask for quotes first | "First quote the supporting sentence, then answer" — inability to find a quote surfaces the gap | 44 |
| Constrain the output | force answers into a fixed schema or list of options | 48 |
| Verify with a second pass | check each claim against the sources | 49 |
| Sample several times | if 5 samples disagree, it's unreliable | 37 |
| Check confidence | low token probabilities correlate with errors | 39 |
| Human review | for anything consequential. Non-negotiable in medicine/law/finance. | 49 |
None of these eliminates hallucination. Layered, they reduce it to acceptable rates for most applications — and knowing that they're layered defences, not cures, is the professional mindset.
Practice (20 min) — provoke and then fix a hallucination
# Use any chat model you have access to (local or API).
# 1. Ask it something obscure and verifiable:
# "What was the exact population of Lucca, Italy in the 1861 census?"
# "List three papers by <a real but obscure researcher> with DOIs."
# Then VERIFY the answer. Note whether it hedged or asserted confidently.
#
# 2. Now give it the source and constrain it:
context = """
Lucca is a city in Tuscany, Italy. According to the guide, the historic centre
is surrounded by intact Renaissance-era walls. The guide does not state
population figures.
"""
prompt = f"""Answer the question using ONLY the context below.
If the context does not contain the answer, reply exactly: "Not stated in the provided text."
Do not use any other knowledge.
Context:
{context}
Question: What was the population of Lucca in the 1861 census?
Answer:"""
# Run this. A well-behaved model now says "Not stated in the provided text."
Write in notes.md: how confident was the model when it was wrong? This
calibration gap is the thing to internalise.
Check yourself
- Give two mechanistic reasons LLMs hallucinate.
- Why are fake citations so convincingly formatted?
- What is the single most effective mitigation?
- Why doesn't the model just say "I don't know"?
Further reading
- Video: Karpathy's "Deep Dive into LLMs" — the hallucination section is excellent and shows how models are trained to say "I don't know".
- Article: "Why Language Models Hallucinate" — search for recent OpenAI and Anthropic research posts on this; the framing has improved a lot recently.
- Paper: "Survey of Hallucination in Natural Language Generation" (Ji et al.) — a thorough taxonomy if you want depth.
- Practical: the RAGAS documentation (docs.ragas.io) on faithfulness metrics — how to actually measure grounding in your own app.
End of Part 8 — Milestone check
- I can state what an LLM computes in one sentence
- I can trace a prediction through the 8 steps of Chapter 32
- I can explain where LLM training labels come from
- I can state the Chinchilla result and why real models over-train
- I can explain the difference between a base and an instruct model
- I have run a base model and an instruct model side by side
- I can explain hallucination mechanistically, not just as "it lies"
You now understand LLMs. The rest of the guide is about using and building them.