Part 6 — The Transformer

This is the most important part of the guide. Take it slowly. Do the arithmetic by hand once — it is the difference between "I've heard of attention" and "I understand attention".

Chapter 22 — The problem: reading a sentence all at once

In one sentence

Instead of reading word by word and remembering (an RNN), read the whole sentence simultaneously and let every word decide which other words matter to it.

Where we are

You have three ingredients from earlier parts:

Embeddings (Part 5): every word is a vector of numbers carrying meaning.
Neural layers (Part 4): matrix multiply + non-linearity, trained by gradient descent.
Attention (Chapter 17): the idea of looking back at everything and focusing on the relevant bits.

The Transformer combines them, deletes the RNN, and that's the architecture.

The problem it solves

Consider this sentence:

"The animal didn't cross the street because it was too tired."

What does "it" refer to — the animal or the street? A human knows instantly: "tired" applies to animals. To resolve this, the representation of "it" must be influenced by "animal" and by "tired" — words 3 positions back and 3 positions forward.

An RNN would have to carry that information through its hidden state, step by step, hoping it survives. A Transformer lets "it" directly look at every other word in one step and pull in what it needs.

The three properties that made it win

Property	RNN	Transformer
Training speed	must process word 1, then 2, then 3... — can't parallelise	processes all words at once — one big matrix multiply on a GPU
Distance	information from 50 words ago is degraded	every word connects directly to every other word
Scaling	hard to make deep or wide	scales smoothly to 100+ layers and trillions of parameters

The speed one was decisive: a Transformer trains in hours what an LSTM needed weeks for. That let researchers try much bigger models on much more data — and that is what produced LLMs.

Check yourself

Why is "it" a hard problem for a model?
What is the single biggest practical advantage of the Transformer over an RNN?