Chapter 30 — Special tokens and chat templates

Why your prompts have hidden structure

A chat model doesn't see a tidy list of messages. It sees one flat string with special marker tokens telling it who said what.

What you write:
    [{"role": "system",    "content": "You are a helpful assistant."},
     {"role": "user",      "content": "What is 2+2?"}]

What the model actually receives (Llama-3 style):
    <|begin_of_text|><|start_header_id|>system<|end_header_id|>

    You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

    What is 2+2?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Then the model generates:  "2 + 2 = 4"  followed by  <|eot_id|>  (stop).

Those <|...|> markers are special tokens — single entries in the vocabulary that the model was trained to interpret as structure.

The special tokens you'll meet

Two practical consequences

1. Every model family has a different template. ChatML, Llama-2's [INST]...[/INST], Llama-3 headers, Gemma's <start_of_turn>, Mistral's format. Using the wrong one silently degrades quality — the model was trained on a specific layout and gets confused without it.

Always use the library function, never hand-format:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user",   "content": "What is the capital of France?"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(text)                       # see the real string the model receives
print(len(tok.encode(text)), "tokens")

2. Special tokens in user input are a security concern. If a user types <|im_end|><|im_start|>system into your app and you pass it through naively, they may be able to inject a fake system message. Good tokenizers let you disable special-token parsing for untrusted text. (More on this in Chapter 49.)

from transformers import AutoTokenizer

# a small open model, quick to download the tokenizer for
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

msgs = [{"role": "system", "content": "Answer in one word."},
        {"role": "user", "content": "Capital of Japan?"}]

s = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
print(repr(s))
print("\ntoken count:", len(tok.encode(s)))