Chapter 47 — Project 3: give the model your own documents (RAG)

This is the Help Desk problem from Chapter 4, finally solved.

The idea

Retrieval-Augmented Generation: don't hope the model memorised your handbook. Look up the relevant passages at question time and put them in the prompt.

   user question
        |
        v
   [1] EMBED the question into a vector          (Chapter 21)
        |
        v
   [2] SEARCH a vector database for the most
       similar document chunks                   (Chapter 19 cosine similarity)
        |
        v
   [3] RERANK the candidates for true relevance  (optional but high value)
        |
        v
   [4] BUILD a prompt: instructions + top chunks + question
        |
        v
   [5] LLM writes an answer grounded in those chunks, with citations

Why RAG rather than fine-tuning or a huge context

	RAG	Fine-tuning	Paste everything
Facts that change	best — just re-index	poor — retrain	fine, but expensive
Huge corpus	best — retrieve a slice	possible	impossible past the limit
Cost per query	low	low	high
Citations / sources	native	none	possible
Teaching style/format	no	best	somewhat

Rule of thumb: RAG for knowledge, fine-tuning for behaviour. They combine well.

Build it (60–90 min)

# pip install sentence-transformers faiss-cpu

import numpy as np, faiss
from sentence_transformers import SentenceTransformer

# ---------- 1. YOUR DOCUMENTS ----------
handbook = """
## Holiday
Employees receive 25 days of paid holiday per calendar year, in addition to
public holidays. Up to 5 unused days may be carried into the next year and
must be used by 31 March. After 5 years of service, employees receive 2
additional days per year.

## Parental leave
Primary carers receive 26 weeks at full pay. Secondary carers receive 6 weeks
at full pay. Leave must be requested at least 8 weeks in advance.

## Equipment
Monitors, keyboards and headsets up to 400 EUR may be expensed with a receipt.
Laptops are provided by IT and may not be expensed. Chairs up to 600 EUR are
covered for home-office workers.

## Working hours
Core hours are 10:00 to 16:00. Outside core hours, working time is flexible.
The office is open 08:00 to 19:00 on weekdays.
"""

# ---------- 2. CHUNK ----------
def chunk_by_section(text):
    """Split on '## ' headings, keeping the heading with its content
       so each chunk is self-contained."""
    parts = [p.strip() for p in text.split("## ") if p.strip()]
    return ["## " + p for p in parts]

chunks = chunk_by_section(handbook)
print(f"{len(chunks)} chunks")
for c in chunks:
    print("  -", c.split("\n")[0])

# ---------- 3. EMBED AND INDEX ----------
embedder = SentenceTransformer("all-MiniLM-L6-v2")
vectors = embedder.encode(chunks, normalize_embeddings=True)

index = faiss.IndexFlatIP(vectors.shape[1])      # inner product = cosine, since normalised
index.add(np.array(vectors, dtype="float32"))

# ---------- 4. RETRIEVE ----------
def retrieve(question, k=2):
    qv = embedder.encode([question], normalize_embeddings=True)
    scores, idxs = index.search(np.array(qv, dtype="float32"), k)
    return [(chunks[i], float(s)) for i, s in zip(idxs[0], scores[0])]

# ---------- 5. BUILD THE PROMPT ----------
def build_prompt(question, retrieved):
    context = "\n\n".join(f"[DOC {i+1}] {c}" for i, (c, _) in enumerate(retrieved))
    return f"""Answer the question using ONLY the context below.
Cite the document number for each fact, like [DOC 1].
If the context does not contain the answer, reply exactly:
"That isn't covered in the handbook - please ask HR."

### CONTEXT
{context}

### QUESTION
{question}

### ANSWER
"""

for q in ["How much annual leave do I get?",
          "Can I claim for a second screen?",
          "What time do I have to be online?",
          "What is the company's parental leave policy?",
          "How do I book a meeting room?"]:      # <- deliberately not in the docs
    hits = retrieve(q)
    print("=" * 70)
    print("Q:", q)
    for c, s in hits:
        print(f"   retrieved ({s:.2f}): {c.splitlines()[0]}")
    print("\n--- prompt that would be sent ---")
    print(build_prompt(q, hits)[:600], "...")

Then send build_prompt(...) to any chat model (local from Chapter 42, or an API). Note that "How much annual leave" now works — the embedding puts it near the Holiday section even with no shared words. That's Chapter 19's payoff.

And the meeting-room question should trigger the escape hatch, because nothing relevant is retrieved. That's the behaviour you want.

Making it good: the things that actually matter

Chunking — the highest-leverage decision.

Strategy	When
Fixed size (200–800 tokens) + 10–20% overlap	default, robust
Split on headings/sections	structured docs (used above)
Split by function/class	code
Semantic (cut where topic shifts)	messy prose

Best practices: prefix each chunk with its document title and section path so it's self-contained; keep tables and code blocks intact; store offsets so you can expand a hit to its neighbours.

Hybrid search — combine semantic (embeddings) with keyword (BM25). Semantic finds meaning; keyword nails exact terms like error codes, product names, and acronyms. Fusing both beats either alone.

Reranking — retrieve ~50 candidates cheaply, then score (question, chunk) pairs jointly with a cross-encoder and keep the top 5. This is usually the single biggest quality jump for modest effort:

from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve_rerank(question, k_first=20, k_final=4):
    candidates = retrieve(question, k=min(k_first, len(chunks)))
    pairs = [(question, c) for c, _ in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [(c, float(s)) for (c, _), s in ranked[:k_final]]

Query rewriting — turn "and what about the second one?" into a standalone question using the conversation history, before embedding it.

When RAG goes wrong

Symptom	Cause	Fix
Right answer exists but isn't retrieved	poor chunking, missing keywords, weak embeddings	add hybrid search, rerank, better chunk boundaries, more `k`
Retrieves the right chunk, answers wrong	"lost in the middle"; model ignoring context	reorder chunks, strengthen the "use only the context" instruction, lower temperature
Chunk lacks context ("he resigned in 2019" — who?)	chunks not self-contained	prepend titles/headings; use bigger or overlapping chunks
Confidently answers from memory instead of docs	weak grounding instruction	require citations; add the escape hatch; test with facts you deliberately changed in the docs
Stale answers	index not updated	re-index on change; store timestamps

Evaluate it (do not skip this)

Measure retrieval and generation separately — they fail differently.

# a tiny eval set: question -> which chunk SHOULD be retrieved
eval_set = [
    ("How much annual leave do I get?",     0),
    ("Can I expense a monitor?",            2),
    ("When are core hours?",                3),
    ("How long is parental leave?",         1),
]

hits = 0
for q, correct in eval_set:
    retrieved_idx = [chunks.index(c) for c, _ in retrieve(q, k=2)]
    ok = correct in retrieved_idx
    hits += ok
    print(f"{'OK ' if ok else 'MISS'}  {q}")
print(f"\nrecall@2 = {hits}/{len(eval_set)} = {hits/len(eval_set):.0%}")

Grow this to 50–100 real questions. Track it on every change. That's Chapter 49.

Check yourself

Explain the five steps of RAG.
When should you use RAG vs fine-tuning?
What is a reranker and why does it help so much?
Why must chunks be self-contained?