Chapter 36 — From your question to the first word

Two phases, with opposite characteristics

YOU SEND:  "Summarise this 3000-word document: ..."

PHASE 1 -- PREFILL                         PHASE 2 -- DECODE
"read the prompt"                          "write the answer"
------------------                         -----------------
Process ALL prompt tokens AT ONCE          Generate ONE token at a time,
in one big forward pass.                   each needing a full pass through
                                           the model.
Fast per token (GPU is fully busy
doing huge matrix multiplies).             Slow per token, and NOT because of
                                           arithmetic -- because the GPU must
Cost grows with prompt length.             READ ALL THE WEIGHTS FROM MEMORY
                                           for every single token.
Determines: TIME TO FIRST TOKEN
                                           Determines: TOKENS PER SECOND

The counter-intuitive bit: decoding is limited by memory, not maths

This surprises everyone, so it's worth stating plainly.

To generate one token, the model must multiply a single vector by every weight matrix. That means reading all 14 GB of a 7B model out of GPU memory — and doing relatively little arithmetic with each number it reads. The processor sits mostly idle, waiting for memory.

Rough single-stream decode speed:

   tokens per second  ~=  (GPU memory bandwidth)  /  (model size in bytes)

   Example: 3,350 GB/s bandwidth,  14 GB model
            = ~240 tokens/second theoretical ceiling (real: ~100-150)

Three big consequences:

1. Smaller/quantised models are faster, almost proportionally — because there's less to read.
2. Batching is nearly free. Serving 32 users at once reads the weights once for all 32, so total throughput goes up ~32x for little extra latency. This is why hosted APIs are cheap and why a private GPU serving one user is inefficient.
3. Long prompts slow the first token, not the rest. A 50,000-token prompt has a slow prefill (high time-to-first-token) but then generates at roughly normal speed.

The two latency numbers you should measure

TTFT  (Time To First Token)   -- how long until text starts appearing.
                                 Dominated by prompt length and queue wait.
                                 This is what "feels slow" to users.

TPOT  (Time Per Output Token) -- how fast text streams after that.
                                 Roughly constant.

total time  =  TTFT  +  TPOT x (number of output tokens)

Streaming exists entirely because of this split: showing tokens as they arrive makes a 6-second response feel fast, because the user sees something after ~0.4 s.

import time
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name); model.eval()

def timed(prompt_words, new_tokens=50):
    prompt = "Repeat after me. " + ("word " * prompt_words)
    ids = tok(prompt, return_tensors="pt").input_ids

    t0 = time.time()
    with torch.no_grad():                       # prefill only (1 new token)
        model.generate(ids, max_new_tokens=1, do_sample=False,
                       pad_token_id=tok.eos_token_id)
    ttft = time.time() - t0

    t0 = time.time()
    with torch.no_grad():
        model.generate(ids, max_new_tokens=new_tokens, do_sample=False,
                       pad_token_id=tok.eos_token_id)
    total = time.time() - t0

    print(f"prompt {ids.shape[1]:5d} tokens | TTFT {ttft:6.2f}s | "
          f"total {total:6.2f}s | ~{new_tokens/(total-ttft):5.1f} tok/s")

for n in [10, 100, 500, 1500]:
    timed(n)