Chapter 37 — Choosing the next word: temperature, top-p, and friends

The setting

The model gives you probabilities for every token (Chapter 32). Now you must pick one. How you pick changes the personality of the output completely — and these are the settings you'll tune most often in practice.

Greedy: always take the most likely

probabilities:  " Paris" 0.89   " Lyon" 0.02   " the" 0.015  ...
greedy picks:   " Paris"

Fully deterministic — same input, same output, every time.
Best for: classification, data extraction, "fix this bug", anything with one right answer.
Bad for: creative writing (bland), and it can get stuck in repetition loops ("the the the").

Temperature: how much randomness

Divide the logits by T before the softmax.

logits:  [3.0, 1.0, 0.5]

T = 0.5  (cold) -> divide by 0.5 -> [6.0, 2.0, 1.0]  -> softmax [0.97, 0.02, 0.01]
T = 1.0  (normal)                  [3.0, 1.0, 0.5]   -> softmax [0.84, 0.11, 0.05]
T = 2.0  (hot)   -> divide by 2.0 -> [1.5, 0.5, 0.25] -> softmax [0.60, 0.22, 0.18]

T -> 0     the distribution collapses onto the top token  (= greedy)
T = 1      the model's own probabilities, untouched
T > 1      flattened: unlikely tokens get a real chance
T >> 1     approaching random gibberish

Mental model: temperature controls the confidence of the distribution, not the model's knowledge. Low temperature isn't "smarter", it's just less willing to take chances.

Top-k: only consider the k best

Keep the 40 highest-probability tokens, zero out the rest, renormalise, sample.

Prevents the model from ever picking something absurd from the long tail. Weakness: k is fixed, but the right number varies. After "The capital of France is" only one token is sensible; after "Once upon a" there are hundreds.

Top-p (nucleus sampling): consider the smallest set that covers p

Sort tokens by probability, keep adding until their total reaches p (e.g. 0.9),
discard the rest, renormalise, sample.

After "The capital of France is":   " Paris" alone is 0.89 -> nucleus = ~1 token
After "Once upon a":                many tokens needed to reach 0.9 -> ~50 tokens

This adapts automatically — narrow when the model is confident, wide when it isn't. It's the most common setting, usually top_p = 0.9 to 0.95.

The others you'll see

Setting	Effect	Typical
min_p	keep tokens with probability >= `min_p x (top token's probability)`	0.05–0.1; a robust modern alternative to top-p
repetition_penalty	divide the logits of already-used tokens by `r > 1`	1.0–1.1 (higher damages quality)
frequency_penalty	subtract a bit for each time a token was used	0–0.5
presence_penalty	subtract a fixed amount if a token appeared at all	0–0.5 (pushes toward new topics)
max_tokens	hard cap on output length	always set this
stop	strings that halt generation	task-specific
seed	makes sampling reproducible	set it for tests

Which settings for which job

Classification / extraction / routing   temperature 0            (deterministic)
Code: fix a bug                          temperature 0
Code: write a function                   temperature 0.1 - 0.3
Factual Q&A, RAG answers                 temperature 0 - 0.3
General chat                             temperature 0.7, top_p 0.9
Brainstorming, fiction                   temperature 0.9 - 1.1, top_p 0.95
Sampling many candidates to compare      temperature 0.7 - 1.0, different seeds

Most "the model is being stupid today" complaints are actually the wrong decoding settings. Repetitive output? You're too close to greedy on an open-ended task. Inconsistent factual answers? Your temperature is too high.

Practice (25 min) — feel the difference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name); model.eval()

def gen(prompt, **kw):
    text = tok.apply_chat_template([{"role":"user","content":prompt}],
                                   tokenize=False, add_generation_prompt=True)
    ids = tok(text, return_tensors="pt").input_ids
    with torch.no_grad():
        out = model.generate(ids, max_new_tokens=60,
                             pad_token_id=tok.eos_token_id, **kw)
    return tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True).strip()

p = "Write one sentence about the sea."

print("GREEDY (run 3x -- identical every time):")
for _ in range(3):
    print("  ", gen(p, do_sample=False))

print("\nTEMPERATURE 0.7 (run 3x -- varies):")
for _ in range(3):
    print("  ", gen(p, do_sample=True, temperature=0.7, top_p=0.9))

print("\nTEMPERATURE 1.5 (run 3x -- wild):")
for _ in range(3):
    print("  ", gen(p, do_sample=True, temperature=1.5, top_p=0.99))

Then:

Ask a factual question ("What is the capital of Japan?") at temperature 1.5, five times. Does it ever get it wrong? This is why factual tasks use temperature 0.
Set temperature=0.01 on the creative prompt and run 5 times. Notice the repetitiveness.
Watch the actual filtering happen:

import torch
ids = tok("The capital of France is", return_tensors="pt").input_ids
with torch.no_grad():
    logits = model(ids).logits[0, -1]

for T in [0.2, 1.0, 2.0]:
    probs = torch.softmax(logits / T, dim=-1)
    top = torch.topk(probs, 5)
    print(f"T={T}: " + "  ".join(
        f"{tok.decode([i])!r}={p:.3f}" for p, i in zip(top.values, top.indices)))

You will see the distribution flatten as T rises.

Check yourself

What does temperature do to the logits, and why does that change randomness?
Why is top-p usually better than top-k?
What settings would you use for a JSON-extraction task? For a poem?