Chapter 18 — The problem with treating words as symbols

One-hot encoding: what we've been doing

Up to now, every word has been a slot in a list:

Vocabulary: [ cat, dog, kitten, car, democracy ]   (imagine 50,000 entries)

cat       = [1, 0, 0, 0, 0]
dog       = [0, 1, 0, 0, 0]
kitten    = [0, 0, 1, 0, 0]
car       = [0, 0, 0, 1, 0]
democracy = [0, 0, 0, 0, 1]

This is called one-hot encoding (one position is "hot"/1, the rest are 0).

Now measure similarity between any two of them with a dot product:

cat . dog       = 1*0 + 0*1 + 0*0 + 0*0 + 0*0 = 0
cat . kitten    = 0
cat . democracy = 0

Every distinct word is exactly equally unrelated to every other word. The representation contains zero information about meaning.

Three concrete consequences

1. No generalisation. Your classifier learns that "excellent" means positive review. It learns nothing about "superb", "outstanding", "fantastic" — each must be learned separately, from its own examples.
2. Huge and wasteful. 50,000-dimensional vectors that are 99.998% zeros.
3. The Chapter 11 failure. "How much annual leave?" scored zero against "25 days of paid holiday" because they share no words. TF-IDF can only match identical strings.

What we want instead

                          A "meaning space"

              pets                              vehicles
                *cat                                *car
             *kitten                              *truck
                *dog                                *bus
                  *puppy
                                       *democracy
                                          *parliament    <- politics

Words with related meanings should be close together. Then:

• "annual leave" lands near "holiday" -> matching works
• learning about "excellent" partly transfers to "superb"
• vectors are ~300 numbers instead of 50,000, and all of them carry information

This is called a dense or distributed representation, or an embedding. Now: where do we get it?

"I adopted a ___ from the shelter"        -> cat, dog, puppy, kitten
"the ___ needs to go to the vet"          -> cat, dog, puppy, kitten
"my ___ knocked over the vase"            -> cat, dog, puppy, kitten

"I drove the ___ to work"                 -> car, truck, van
"the ___ needs an oil change"             -> car, truck, van