Chapter 11 — Turning words into numbers: bag-of-words and TF-IDF

In one sentence

Count words per document, then down-weight words that appear everywhere.

The problem with plain counting

In your spam filter, "the" appears in every email — spam and ham alike. Counting it adds noise, and long documents get big counts for everything just by being long.

The idea: TF-IDF

Two adjustments:

TF (term frequency) — how often the word appears in this document. Common word here = probably important here.
IDF (inverse document frequency) — how rare the word is across all documents. A word in every document tells you nothing; a word in 2% of documents is highly informative.

tf-idf(word, doc) = tf(word, doc)  x  idf(word)

idf(word) = log( total number of documents / number of documents containing word )

Worked example, with numbers

You have 1,000 documents.

word	appears in this doc	appears in N docs	idf = log(1000/N)	tf-idf
"the"	12 times	1000 docs	log(1) = 0	12 × 0 = 0
"meeting"	3 times	200 docs	log(5) = 1.61	3 × 1.61 = 4.83
"viagra"	2 times	5 docs	log(200) = 5.30	2 × 5.30 = 10.6

"the" contributes nothing; "viagra" dominates. Exactly what you want, and it happened automatically from statistics — no stop-word list needed.

Practice (15 min)

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

docs = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "the cat chased the dog",
    "quantum entanglement in the lab",
]

vec = TfidfVectorizer()
X = vec.fit_transform(docs)
words = vec.get_feature_names_out()

import pandas as pd
print(pd.DataFrame(X.toarray().round(2), columns=words))

Look at the column for the: near-zero everywhere. Look at quantum: high in document 4 only. You just watched TF-IDF find what matters.

Now compute document similarity — this is the seed of search and of RAG:

from sklearn.metrics.pairwise import cosine_similarity
sim = cosine_similarity(X)
print(np.round(sim, 2))
# docs 0 and 1 should be most similar (both "sat on the")
# doc 3 should be dissimilar to everything

Applying it to the Help Desk problem

handbook = [
    "Employees receive 25 days of paid holiday per calendar year.",
    "Parental leave is 26 weeks at full pay for the primary carer.",
    "Monitors and keyboards up to 400 EUR may be expensed with a receipt.",
    "The office is open from 8am to 7pm on weekdays.",
]

vec = TfidfVectorizer()
H = vec.fit_transform(handbook)

def ask(question, k=1):
    q = vec.transform([question])
    scores = cosine_similarity(q, H)[0]
    best = scores.argsort()[::-1][:k]
    for i in best:
        print(f"  {scores[i]:.2f}  {handbook[i]}")

ask("How many holiday days do I get?")       # works
print()
ask("How much annual leave do I have?")      # ??? try it

Run it. The second question probably fails or scores near zero, because "annual leave" shares no words with "holiday". TF-IDF matches words, not meaning.

That failure is the exact motivation for Part 5 (word vectors). Keep this script; you'll fix it there.

Common confusions

TF-IDF is not machine learning — it's a fixed formula computed from counts. It's a feature extraction step that feeds a model.
BM25 is a refined TF-IDF used by search engines (Elasticsearch, Lucene). You'll meet it again in Part 11 as the "keyword" half of hybrid search.

Check yourself

Why does "the" get a TF-IDF near zero?
What does cosine similarity measure between two documents?
Why did "How much annual leave" fail?