Chapter 42 — Project 2: fine-tune a real model with LoRA

Why fine-tune instead of pretrain

Pretraining taught the model language. Fine-tuning teaches it your task, format, or style — and needs a few thousand examples instead of trillions.

Fine-tuning is good for: output format, tone/persona, a domain's vocabulary and conventions, task-specific behaviour, and making a small model do one job as well as a big general model.

Fine-tuning is bad for: injecting facts (unreliable; use RAG — Chapter 47) and anything that changes weekly.

What LoRA is

Full fine-tuning updates all 8 billion parameters — needing many times the model size in GPU memory. LoRA (Low-Rank Adaptation) freezes the original weights and trains a small "patch" beside them:

   original weight matrix W  (frozen, e.g. 4096 x 4096 = 16.7M numbers)
                +
   a small update:  B x A    where A is 8 x 4096 and B is 4096 x 8
                             = only 65,536 numbers  (0.4% as many!)

   effective weight = W + (B x A)

Consequences:

Trains on far less memory. With QLoRA, short sequence lengths, and the right runtime, many 7B-class models can be fine-tuned on a single consumer or free Colab-style GPU; plain LoRA for 7B still depends heavily on GPU memory.
The saved adapter is ~10–100 MB instead of 14 GB.
You can keep many adapters for one base model and swap them per task.
Quality is close to full fine-tuning for most instruction/style tasks.

QLoRA goes further: it loads the frozen base model in 4-bit precision, letting you fine-tune much larger models on one consumer GPU.

Step 1 — Prepare your data

The format is a list of conversations. Aim for 500–5,000 high-quality examples — quality and diversity matter far more than volume.

# make_data.py
import json

examples = [
    {"messages": [
        {"role": "user", "content": "How many holiday days do I get?"},
        {"role": "assistant", "content": "You receive 25 days of paid holiday per calendar year, plus public holidays."}]},
    {"messages": [
        {"role": "user", "content": "What's the parental leave policy?"},
        {"role": "assistant", "content": "Primary carers receive 26 weeks at full pay. Secondary carers receive 6 weeks at full pay."}]},
    # ... hundreds more, covering the variety of real questions
]

with open("train.jsonl", "w") as f:
    for ex in examples:
        f.write(json.dumps(ex) + "\n")

Data quality checklist:

Cover the variety of phrasings real users produce (remember your Chapter 4 list).
Include the edge cases and the "I don't know" responses you want.
Be consistent in format — the model learns your format precisely, including your mistakes.
Hold out ~10% as a validation set.
Deduplicate.

Step 2 — Fine-tune

Run this on a free Google Colab GPU (Runtime → Change runtime type → T4 GPU).

# pip install -U transformers peft trl datasets accelerate bitsandbytes

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

BASE = "Qwen/Qwen2.5-0.5B-Instruct"       # start small; scale up once it works

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")

dataset = load_dataset("json", data_files="train.jsonl", split="train")

peft_config = LoraConfig(
    r=16,                  # rank: how big the patch is (8-64 typical)
    lora_alpha=32,         # scaling, usually 2 x r
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
    args=SFTConfig(
        output_dir="./out",
        num_train_epochs=3,             # 1-3 for fine-tuning; more overfits
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,  # effective batch = 16
        learning_rate=2e-4,             # LoRA likes higher LR than full FT
        logging_steps=10,
        save_strategy="epoch",
        fp16=True,                      # T4-class Colab GPUs usually support fp16, not bf16
        max_seq_length=1024,
    ),
)

trainer.train()
trainer.save_model("./my-adapter")      # only ~20 MB!

Step 3 — Use it

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

BASE = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
base = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")
model = PeftModel.from_pretrained(base, "./my-adapter")     # attach the patch
model.eval()

def ask(q):
    text = tok.apply_chat_template([{"role":"user","content":q}],
                                   tokenize=False, add_generation_prompt=True)
    ids = tok(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**ids, max_new_tokens=120, do_sample=False,
                             pad_token_id=tok.eos_token_id)
    return tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True)

print(ask("How much annual leave do I get?"))

Always compare against the base model on the same questions. If fine-tuning didn't clearly help, you need better data, not more epochs.

Key settings and what they do

Setting	Meaning	Typical
`r` (rank)	size of the LoRA patch — capacity	8–64 (start 16)
`lora_alpha`	scaling of the patch	`2 x r`
`learning_rate`	LoRA tolerates higher LR than full fine-tuning	1e-4 – 3e-4
`num_train_epochs`	passes over your data	1–3; more overfits fast
`target_modules`	which layers get patched	attention + MLP projections
`max_seq_length`	truncation length	fit your longest example

Common problems

Symptom	Cause	Fix
Model repeats training examples verbatim	overfitting	fewer epochs, more/varied data, lower `r`
No visible change	LR too low, too few steps, or `target_modules` wrong	raise LR, check the adapter actually loaded
It got worse at everything else	catastrophic forgetting	fewer epochs, lower LR, mix in general examples
Weird formatting in outputs	wrong chat template or inconsistent training data	use `apply_chat_template`; audit your data
Out of memory	model/batch too big	smaller model, `batch_size=1`, more grad accumulation, QLoRA 4-bit

Check yourself

What does LoRA freeze and what does it train?
Why is fine-tuning bad at injecting facts?
What are two signs you've overfitted a fine-tune?