Chapter 34 — Making it an assistant (SFT and RLHF/DPO)

Step 1 — Supervised Fine-Tuning (SFT)

Collect examples of an assistant behaving well:

{"instruction": "Explain photosynthesis to a 10-year-old.",
 "response": "Photosynthesis is how plants make their own food..."}

{"instruction": "Write a Python function to reverse a string.",
 "response": "def reverse(s):\n    return s[::-1]"}

{"instruction": "How do I pick a lock on a car that isn't mine?",
 "response": "I can't help with that. If you're locked out of your own car..."}

Format them with the chat template (Chapter 30) and train with exactly the same next-token loss as pretraining — but compute the loss only on the assistant's reply, not on the prompt. (The prompt is context to condition on, not something to learn to generate.)

Sources of this data: human writers, curated datasets (FLAN, OpenAssistant, UltraChat, Tulu), and increasingly generated by a stronger model and filtered.

Effect: the model learns the shape of being helpful. Most of the "wow, it follows instructions now" comes from this step.

Step 2a — RLHF (the original ChatGPT recipe)

Reinforcement Learning from Human Feedback. Three sub-steps:

1. COLLECT COMPARISONS
   Show a human one prompt and 2-9 model answers. They rank them best to worst.
   (Ranking is much easier and more consistent than writing ideal answers.)

2. TRAIN A REWARD MODEL
   A separate model that reads (prompt, answer) and outputs a score.
   Trained so that answers humans preferred get higher scores.
   -> now you have an automatic judge

3. OPTIMISE THE LLM AGAINST THE JUDGE
   Let the LLM generate answers, score them with the reward model,
   and adjust the LLM to get higher scores -- using reinforcement learning (PPO).

   CRUCIAL EXTRA TERM: a penalty for drifting too far from the SFT model.
   Without it, the model finds nonsense that games the reward model
   ("reward hacking") -- e.g. producing endless flattery because the judge
   rewards agreeableness.

Step 2b — DPO (the simpler modern alternative)

Direct Preference Optimization (2023) noticed that you can skip the separate reward model and the reinforcement-learning loop entirely, and get the same objective as an ordinary classification loss on preference pairs:

For each (prompt, chosen answer, rejected answer):
    make the CHOSEN answer more likely
    make the REJECTED answer less likely
    ... while staying close to the original SFT model

One loss, one model, no rollouts, far more stable. DPO and its variants are now the default for most open models. RLHF/PPO is still used at the frontier and for some settings.

Step 2c — RL from verifiable rewards (how "reasoning models" work)

For tasks with a checkable answer — maths with a known result, code with unit tests — you don't need a learned judge at all. The reward is simply 1 if correct, else 0.

Let the model generate long chains of reasoning, keep what lands on correct answers, and reinforce those trajectories. Over many rounds the model learns to think longer and more usefully before answering. This is the mechanism behind models that visibly "reason" before responding.

Practice (10 min) — see the difference yourself

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

def try_model(name, prompt, chat=False):
    tok = AutoTokenizer.from_pretrained(name)
    model = AutoModelForCausalLM.from_pretrained(name); model.eval()
    if chat:
        text = tok.apply_chat_template(
            [{"role": "user", "content": prompt}],
            tokenize=False, add_generation_prompt=True)
    else:
        text = prompt
    ids = tok(text, return_tensors="pt").input_ids
    with torch.no_grad():
        out = model.generate(ids, max_new_tokens=60, do_sample=False,
                             pad_token_id=tok.eos_token_id)
    print(f"\n=== {name} ===")
    print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

q = "What is the capital of France?"
try_model("Qwen/Qwen2.5-0.5B", q, chat=False)                # BASE model
try_model("Qwen/Qwen2.5-0.5B-Instruct", q, chat=True)        # INSTRUCT model

The base model will ramble or produce more questions. The instruct model will answer. Same architecture, same size, same pretraining — the difference is entirely post-training. This is one of the most illuminating experiments in the guide.