Part 8 — What an LLM actually is

Attention (Part 6), tokens (Part 7), and the training loop you've watched since Chapter 10 — everything you've learned now assembles into one object. The claim this part makes is almost insultingly simple to state and much bigger than it sounds once you see what falls out of it.

Chapter 31 — An LLM is a next-token guesser (that's genuinely it)

In one sentence

An LLM is a big Transformer that, given some text, outputs a probability for every possible next token — and everything else it appears to do follows from that.

The claim, stated plainly

                 +-------------------------------+
   your text --> |    a very large Transformer    | --> a probability for EVERY
   (as tokens)   |    (billions of parameters)    |     token in the vocabulary
                 +-------------------------------+

Given "The capital of France is", the model outputs something like:

   " Paris"      0.89
   " the"        0.03
   " located"    0.02
   " a"          0.01
   " Lyon"       0.004
   ... 100,000 more tokens, each with a tiny probability ...

Then a sampler (Chapter 37) picks one. That token is appended to the text, and the whole thing runs again. And again. That loop is text generation.

There is no separate module for reasoning, or facts, or translation, or code. There is one next-token distribution, and all of those behaviours are consequences of it.

Why this simple objective produces so much

To predict the next token well, across all human text, a model is forced to learn:

To predict the next token in...	it must learn
`"The cat sat on the ___"`	grammar and common sense
`"The capital of Australia is ___"`	facts
`"2 + 2 = ___"`	arithmetic
`"def add(a, b):\n return ___"`	code semantics
`"Translate to French: hello -> ___"`	translation
`"She was furious because he had ___"`	psychology, narrative, causality
`"Q: ...\nA: Let's think step by step. First, ___"`	reasoning patterns

The objective is trivially simple. The competence required to minimise it is enormous. That is the central bet of the LLM era, and it paid off far better than almost anyone expected in 2018.

What an LLM is not

Being precise here saves you from a lot of confusion:

It has no memory between conversations. Each API call is stateless. What feels like memory is your application re-sending the conversation history every time.
It has no goals or intentions. It is a function from text to a probability distribution. Any apparent "wanting" is a pattern in the text it was trained on.
It cannot access the internet, your files, or a calculator unless you build tools and wire them in (Chapter 48).
It doesn't "look things up." Facts are baked into the weights during training; there is no database inside.
It doesn't know what it doesn't know by default — hence hallucination (Chapter 35).

Debated: whether next-token prediction at scale constitutes "understanding" is genuinely contested among serious researchers. One camp says it's sophisticated pattern-matching without meaning ("stochastic parrots"); another says predicting text well requires building internal world-models, and points to interpretability results showing models represent things like board states and spatial relationships. You do not have to settle this to use them well. Operationally: treat it as an extremely capable pattern engine that is often right and confidently wrong in predictable ways.

Check yourself

What does an LLM output, exactly?
Where does an LLM's apparent "memory" of your conversation come from?
Why does one simple objective produce so many abilities?