Chapter 20 — GloVe, FastText, and the family

Three variations worth knowing, because you'll see the names constantly.

GloVe — counting instead of predicting

word2vec slides a window and predicts. GloVe (Stanford, 2014) instead builds one giant table of "how often does word A appear near word B?" across the entire corpus, then finds vectors whose dot products reproduce those counts:

                  the   cat   dog   vet   car
          the      -    120   115    8    95
          cat     120    -    45     30    3
          dog     115   45     -     35    5
          vet      8    30     35     -    1
          car      95    3     5      1    -

Find vectors so that:   vector(cat) . vector(vet)  ~  log(count(cat, vet))

The intuition: ratios of co-occurrence carry meaning. "ice" appears near "solid" much more than "steam" does; that ratio is the signal. GloVe and skip-gram produce similar-quality vectors — GloVe uses global statistics directly, word2vec streams local windows.

FastText — words made of pieces

FastText (Facebook, 2016) represents a word as the sum of its character n-grams:

"where"  ->  <wh, whe, her, ere, re>   plus the whole word
vector("where") = sum of the vectors of those pieces

Three big wins:

1. No unknown words ever. Never seen "unfriendliness"? Compose it from un + friend + li + ness pieces. Great for typos, slang, hashtags.
2. Morphology for free. "run", "runs", "running", "runner" share pieces, so they get related vectors automatically.
3. Works for morphologically rich languages (Finnish, Turkish, German compounds) where word-level vectors fail badly.

Remember this idea — "a word is built from smaller pieces". It is exactly what tokenization does for LLMs (Part 7), for exactly the same reasons.

Quick comparison

import gensim.downloader as api
w2v   = api.load("word2vec-google-news-300")     # ~1.6 GB, skip if slow
glove = api.load("glove-wiki-gigaword-100")

for w in ["computer", "happy", "london"]:
    print(f"\n{w}:")
    print("  glove:", [x for x,_ in glove.most_similar(w, topn=5)])
    print("  w2v  :", [x for x,_ in w2v.most_similar(w, topn=5)])