Chapter 19 — word2vec: meaning from company

The two variants

Sentence:  "the quick brown fox jumps over the lazy dog"
Focus word: "fox"        Context window: 2 words each side
Context = {quick, brown, jumps, over}

SKIP-GRAM                            CBOW (Continuous Bag of Words)
---------                            ------------------------------
given the CENTER word,               given the CONTEXT words,
predict each CONTEXT word            predict the CENTER word

   "fox" --> quick                   quick \
   "fox" --> brown                   brown  \
   "fox" --> jumps                          |--> "fox"
   "fox" --> over                    jumps  /
                                     over  /

slower, better on rare words         faster, better on frequent words
(the usual choice)

1. Start with random vectors for every word (say 300 numbers each).

2. Slide a window through the text. For "fox" with neighbour "brown":
     - compute a score = dot product of  vector(fox) . vector(brown)
     - we WANT this score to be high (they really do co-occur)

3. Also pick a few RANDOM words that were NOT nearby ("negative samples"),
   e.g. "democracy", "toaster", "algebra":
     - we WANT  vector(fox) . vector(democracy)  to be LOW

4. Nudge all these vectors to make the true pair more similar and the random
   pairs less similar. (Gradient descent, Chapter 10.)

5. Repeat for billions of word pairs.

Why "negative sampling" was the crucial trick

The naive version asks: "out of all 1,000,000 words in the vocabulary, what's the probability the neighbour is 'brown'?" That requires computing a score for every word in the vocabulary, for every training pair. With billions of pairs, that's impossible.

Negative sampling replaces one impossible 1,000,000-way question with ~6 easy yes/no questions:

Instead of:  "which of 1,000,000 words comes next?"     (cost: 1,000,000)
Ask:         "is 'brown' a real neighbour of 'fox'?  yes"     (cost: 1)
             "is 'democracy'?  no"                            (cost: 1)
             "is 'toaster'?    no"                            (cost: 1)
             ... 5 negatives total                            (cost: 6)

That's roughly a 100,000x speedup, and it's why word2vec could train on 100 billion words on ordinary hardware in 2013.

(Two details from the paper that matter in practice: negatives are drawn with probability proportional to word frequency raised to the power 0.75, which samples rare words slightly more often than pure frequency would; and very frequent words like "the" are randomly subsampled out, which both speeds training and improves the vectors.)

The famous result: vector arithmetic

Once trained, relationships become directions in the space:

   vector("king") - vector("man") + vector("woman")  ~=  vector("queen")

   vector("Paris") - vector("France") + vector("Italy")  ~=  vector("Rome")

   vector("walking") - vector("walked") + vector("swam")  ~=  vector("swimming")

Visually:

          queen *----------------* king
                |                |          the "gender" direction is
                |                |          roughly the same vector
                |                |          everywhere in the space
          woman *----------------* man

Nobody designed this. It emerged from "predict the neighbours".

Simplified: the analogy result is real but was over-hyped. It works well for some relations (capitals, gender, tense) and poorly for others, and the standard evaluation excludes the input words from the candidate answers, which flatters it. The durable lesson is the shape of the idea — directions in embedding space carry meaning — which does carry over to modern LLMs.

The uncomfortable part: bias

The vectors learn whatever associations exist in the text, including harmful ones:

vector("man") - vector("woman")  points in a similar direction as
vector("computer_programmer") - vector("homemaker")

This is not a bug in the algorithm; it is a faithful reflection of the training text. It matters because embeddings feed real decisions (CV screening, search ranking). LLMs inherit this, which is a major reason the alignment stage (Chapter 34) and evaluation (Chapter 49) exist.

from gensim.models import Word2Vec
from gensim.utils import simple_preprocess
import gensim.downloader as api

# ---- Option A: train your own on a book (5-10 min) ----
text = open("book.txt", encoding="utf-8", errors="ignore").read()
sentences = [simple_preprocess(line) for line in text.split(".") if len(line) > 20]

model = Word2Vec(
    sentences,
    vector_size=100,   # how many numbers per word
    window=5,          # context radius
    min_count=3,       # ignore words appearing < 3 times
    sg=1,              # 1 = skip-gram, 0 = CBOW
    negative=10,       # negative samples
    epochs=20,
    workers=4,
)
print("vocabulary size:", len(model.wv))
print(model.wv.most_similar("king", topn=8))

# ---- Option B: use pre-trained vectors (better results, ~100 MB download) ----
wv = api.load("glove-wiki-gigaword-100")     # 400k words, 100 dims

print(wv.most_similar("king", topn=8))
print(wv.most_similar("python", topn=8))     # note: snake OR language?
print()
print("cat  vs dog :", round(wv.similarity("cat", "dog"), 3))
print("cat  vs car :", round(wv.similarity("cat", "car"), 3))
print()
print("king - man + woman =", wv.most_similar(positive=["king","woman"],
                                              negative=["man"], topn=3))
print("paris - france + italy =", wv.most_similar(positive=["paris","italy"],
                                                  negative=["france"], topn=3))
print()
print("odd one out:", wv.doesnt_match(["breakfast","lunch","dinner","france"]))

Now fix the Help Desk problem

import numpy as np
import gensim.downloader as api
wv = api.load("glove-wiki-gigaword-100")

handbook = [
    "Employees receive 25 days of paid holiday per calendar year.",
    "Parental leave is 26 weeks at full pay for the primary carer.",
    "Monitors and keyboards up to 400 EUR may be expensed with a receipt.",
    "The office is open from 8am to 7pm on weekdays.",
]

def embed(sentence):
    """Average the word vectors -- crude but surprisingly effective."""
    words = [w for w in simple_preprocess(sentence) if w in wv]
    return np.mean([wv[w] for w in words], axis=0) if words else np.zeros(100)

H = np.array([embed(s) for s in handbook])

def cosine(a, B):
    return (B @ a) / (np.linalg.norm(B, axis=1) * np.linalg.norm(a) + 1e-9)

def ask(q):
    scores = cosine(embed(q), H)
    i = scores.argmax()
    print(f"Q: {q}\n   -> ({scores[i]:.2f}) {handbook[i]}\n")

ask("How many holiday days do I get?")
ask("How much annual leave do I have?")     # <-- the one TF-IDF failed on
ask("Can I claim for a second screen?")     # <-- no shared words at all

The second and third questions should now find the right paragraph, even though they share almost no words with it. You have just built a semantic search engine — the retrieval half of RAG (Part 11).