Chapter 21 — The limit of static vectors, and what came next

The problem: one vector per word is not enough

"I sat on the river bank."            bank = the edge of a river
"I deposited cash at the bank."        bank = a financial institution
"The plane will bank to the left."     bank = to tilt

word2vec must give "bank" one vector. That vector is an average of all senses — a blur that is wrong in every specific case. You saw this in the Practice: most_similar("python") mixes snakes and programming.

Static vectors also cannot capture:

Word order: "dog bites man" and "man bites dog" average to the same vector.
Negation: "not good" averages to something near "good".
Long-range structure: who "she" refers to three sentences ago.

The fix: make the vector depend on the sentence

Contextual embeddings. The vector for "bank" should be computed from the whole sentence, so it's different in each case.

ELMo (2018) did it first: train a bidirectional LSTM language model (predict the next word left-to-right, and the previous word right-to-left) on a big corpus. Then a word's embedding is taken from the LSTM's internal state at that position — which has read the whole sentence. Plugging ELMo vectors into existing models improved essentially every NLP task at once.

That was the proof that pretraining a language model and reusing its internals beats training from scratch per task — the idea that everything after is built on.

The through-line to LLMs

one vector per word            word2vec / GloVe        2013-2014
        |
   + built from character pieces     FastText          2016
        |
   + depends on the sentence         ELMo (biLSTM)     2018
        |
   + deep, bidirectional, Transformer   BERT           2018
        |
   + generative, in-context learning    GPT-2/3 ...    2019+

An LLM's internal activations are contextual embeddings. When you call an "embeddings API" for search in Part 11, you are using the 2024 descendant of ELMo's 2018 idea — a Transformer producing a vector for a whole sentence, in context.

Practice (15 min) — see contextual embeddings work

from sentence_transformers import SentenceTransformer
import numpy as np

m = SentenceTransformer("all-MiniLM-L6-v2")     # small, fast, 384 dims

sents = [
    "I sat on the river bank watching the water.",
    "The bank approved my mortgage application.",
    "We fished from the muddy bank all afternoon.",
    "Interest rates at the bank went up again.",
]
E = m.encode(sents, normalize_embeddings=True)

print("similarity matrix (rows/cols = sentences above):")
print(np.round(E @ E.T, 2))

You should see sentences 0 and 2 (river banks) score high with each other, and 1 and 3 (money banks) score high with each other — the same word, correctly separated by context. Static word vectors cannot do this.

Check yourself

Give three things a single fixed vector per word cannot capture.
What is a contextual embedding?
Why was ELMo important historically?

End of Part 5 — Milestone check

I can explain why one-hot vectors can't express similarity
I can state the distributional hypothesis
I can explain what word2vec trains on and what it keeps
I can explain why negative sampling was necessary
I have run word vectors and found analogies myself
I can explain why static vectors aren't enough

You are now ready for the Transformer. Everything you need is in place: loss and gradients, neural layers, embeddings, and the idea of attention from Chapter 17.