Chapter 32 — Walking through one prediction, end to end

Let's trace "The capital of France is" through the machine, with shapes.

STEP 1: TOKENIZE
   "The capital of France is"
   -> [791, 6864, 315, 9822, 374]              5 token IDs

STEP 2: EMBEDDING LOOKUP
   Each ID indexes a row of the embedding matrix E [vocab x d_model]
   -> a [5 x 4096] matrix    (5 tokens, each now a 4096-number vector)

   Think: each token has been turned into its "meaning coordinates".

STEP 3: THE STACK OF TRANSFORMER BLOCKS  (repeat 32 times)

   for each block:
       (a) NORMALISE the vectors
       (b) ATTENTION: every token looks at every earlier token and
           pulls in relevant information
           -- "France" enriches "is" with country-ness
           -- "capital" enriches "is" with the capital-of relation
       (c) ADD the result back to the running vectors (residual)
       (d) NORMALISE again
       (e) FEED-FORWARD: each token's vector is processed on its own
           -- this is where stored facts get retrieved and mixed in
       (f) ADD back again (residual)

   Shape stays [5 x 4096] the whole way through. The NUMBERS change:
   each token's vector accumulates more and more context.

STEP 4: TAKE THE LAST POSITION
   For generating the next token, only the LAST token's vector matters --
   it has absorbed information from all the others.
   -> a single [4096] vector

STEP 5: PROJECT TO VOCABULARY
   Multiply by the output matrix [4096 x 128000]
   -> [128000] numbers, one per vocabulary token. These are "LOGITS":
      raw scores, can be any value, positive or negative.

      logits[" Paris"] =  12.4
      logits[" Lyon"]  =   6.1
      logits[" the"]   =   5.8
      logits[" banana"] = -3.2
      ...

STEP 6: SOFTMAX -> PROBABILITIES
   -> [128000] probabilities summing to 1.0

      " Paris"  0.89
      " Lyon"   0.02
      " the"    0.015
      ...

STEP 7: SAMPLE
   Pick one token according to those probabilities (Chapter 37).
   -> " Paris"

STEP 8: LOOP
   Append it. Now the input is "The capital of France is Paris".
   Run steps 1-7 again for the next token.
   Stop when the model produces an end-of-text token, or you hit a limit.

The cost, in one number

A useful rule of thumb: producing one token costs roughly 2 x (number of parameters) arithmetic operations.

7-billion-parameter model:  ~14 billion operations PER TOKEN generated

A modern GPU does trillions of operations per second, so this takes milliseconds — but multiply by 500 tokens of output, by thousands of users, and you see where the cost comes from.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

name = "Qwen/Qwen2.5-0.5B"           # small: ~1 GB download, runs on CPU
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
model.eval()

def next_token_table(prompt, k=10):
    ids = tok(prompt, return_tensors="pt")
    with torch.no_grad():
        out = model(**ids)
    logits = out.logits[0, -1]                  # last position only  (step 4-5)
    probs = torch.softmax(logits, dim=-1)       # (step 6)
    top = torch.topk(probs, k)
    print(f"\nPrompt: {prompt!r}")
    print(f"  {'token':<20} {'prob':>7}  {'logit':>7}")
    for p, i in zip(top.values, top.indices):
        print(f"  {tok.decode([i])!r:<20} {p.item():7.4f}  {logits[i].item():7.2f}")

next_token_table("The capital of France is")
next_token_table("2 + 2 =")
next_token_table("Once upon a")
next_token_table("def fibonacci(n):\n    if n <")
next_token_table("The capital of Australia is")     # a trickier fact

Generate text manually (the loop, spelled out)

def generate(prompt, n=30, temperature=0.8):
    ids = tok(prompt, return_tensors="pt").input_ids
    for _ in range(n):
        with torch.no_grad():
            logits = model(ids).logits[0, -1] / temperature
        probs = torch.softmax(logits, dim=-1)
        nxt = torch.multinomial(probs, 1)          # sample one token
        ids = torch.cat([ids, nxt.unsqueeze(0)], dim=1)
        if nxt.item() == tok.eos_token_id:
            break
    return tok.decode(ids[0])

print(generate("The three most important things in life are"))

That is model.generate() with the lid off. Nothing else is happening.