Chapter 43 — Teaching preferences (a taste of DPO)

Optional but instructive: this is Chapter 34's alignment step, hands-on.

The data

Instead of "here's the right answer", you supply "this answer is better than that one":

prefs = [
  {"prompt": "How many holiday days do I get?",
   "chosen":   "You receive 25 days of paid holiday per year, plus public holidays.",
   "rejected": "I think it's around 20-30 days, but you should check with HR."},
  {"prompt": "Can I expense a monitor?",
   "chosen":   "Yes. Monitors up to 400 EUR can be expensed with a receipt.",
   "rejected": "Sure, expense whatever you need."},
]

Notice what the pairs teach: be specific, don't hedge unnecessarily, don't invent permissions. Preferences capture taste that's hard to write as a rule.

The training

from trl import DPOTrainer, DPOConfig
from datasets import Dataset

ds = Dataset.from_list(prefs)

trainer = DPOTrainer(
    model=model,                 # your SFT'd model
    ref_model=None,              # TRL uses a frozen copy automatically
    args=DPOConfig(
        output_dir="./dpo-out",
        beta=0.1,                # how strongly to stay near the original model
        learning_rate=5e-6,      # MUCH lower than SFT
        num_train_epochs=1,
        per_device_train_batch_size=2,
    ),
    train_dataset=ds,
    processing_class=tok,
)
trainer.train()

Key settings: beta around 0.1 (higher = more conservative), learning rate around 5e-6 (10–50x lower than SFT), 1 epoch. DPO is easy to overdo — if outputs become degenerate, your LR is too high or beta too low.

Where preference data comes from

1. Humans ranking outputs — best quality, slow, expensive.
2. A stronger model as judge — "which answer is better?" (called RLAIF). Cheap and scalable; inherits the judge's biases.
3. Automatic checks — for maths/code, "did it pass the tests?" is a perfect, free preference signal.
4. Real user feedback — thumbs up/down in your product. The best long-term source; instrument for it from day one.