Chapter 65 — Project ideas by level

Beginner (consolidate this guide)

1. Train a tiny LLM on your own writing; compare it to one trained on a novel.
2. Build a semantic search engine over your notes/bookmarks.
3. Compare 5 tokenizers on text in 3 languages; write up the fairness gap.
4. A "which model should I use?" cost/latency/quality calculator.
5. A CLI that summarises any URL, with token counting and a cost readout.

Intermediate

6. RAG over a document collection you care about — with reranking, citations, and a 100-question eval set.
7. Fine-tune a small model for one narrow task; prove it beats a bigger general model on that task, at lower cost.
8. A prompt-injection test harness: collect payloads, run them against your app in CI.
9. An LLM-as-judge evaluation pipeline with bias controls (position swapping, length normalisation).
10. A router that sends easy queries to a cheap model and hard ones to a strong model; measure the savings.

Advanced

11. Reproduce GPT-2 (124M) following Karpathy's video; document every deviation.
12. Implement FlashAttention-style tiled attention yourself and benchmark it.
13. Build a DPO pipeline: generate candidates, collect preferences, train, evaluate before/after.
14. Quantize a model to 4-bit and measure the quality/speed/memory trade-off on your own eval set.
15. An agent with tools, sandboxing, human-in-the-loop approval, full tracing, and a red-team suite.
16. A long-running outer-loop agent (Chapter 79's shape) for a different job, such as fixing flaky tests or burning down lint debt, with pass^k measured on 20 historical tasks before it's switched on.
17. Compare three decision approaches (a local LLM's yes/no probabilities, a fine-tuned small classifier, a hosted decision model) on the same 300 labelled cases: calibration, escalation rate, latency, cost.