Part 10 — Build your own LLM
Two projects. The first trains a language model from nothing. The second adapts a real open model to your own data. Both run on modest hardware.
Chapter 40 — What you can realistically build
Be honest about scale
| Your hardware | Realistic model | Training data | Time | Result |
|---|---|---|---|---|
| Laptop CPU | ~0.2–2M params, character-level | 1–10 MB | minutes | Shakespeare-shaped text. Perfect for learning. |
| Laptop with GPU / free Colab | 10–125M params | 100 MB – 2 GB | hours | Coherent short sentences; a real, tiny LLM |
| 1 rented GPU (~$1–3/hour) | 125M – 1B | 5–50 GB | 1–3 days | GPT-2-class. Usable for a narrow task after fine-tuning. |
| 8 GPUs for a month | 1–8B | 100B+ tokens | weeks | A small but genuinely capable base model |
| Thousands of GPUs, months | 8B – 1T+ | 10–20T tokens | months | Frontier. $10M–$100M+. Not you, not today. |
The honest advice: train a tiny model once, to understand it. Then, for anything real, fine-tune an existing open model — you get 90%+ of the value for <0.1% of the cost, because someone else already paid for the pretraining.
The two projects
- Project 1 (Chapter 41): train a small GPT from scratch on a book. You'll understand every line.
- Project 2 (Chapter 42): fine-tune a real open model on your own data using LoRA, on a free Colab GPU.
Further reading
- Book: Sebastian Raschka, Build a Large Language Model (From Scratch) — an entire book on Project 1, done carefully. The best purchase for this stage.
- Code: github.com/karpathy/nanoGPT — the reference implementation.
- Video: Karpathy, "Let's reproduce GPT-2 (124M)" (~4 h) — a complete, real training run, narrated.