Chapter 9 — Your first model: is this email spam?

In one sentence

Turn each email into a list of word counts, learn a weight for each word, add them up, and squash the total into a probability.

The idea, in plain language

Imagine scoring emails by hand. You'd give points:

"free"      +3 points   (spammy)
"viagra"    +8 points
"meeting"   -4 points   (legit)
"invoice"   -2 points
"click"     +2 points

Add the points for the words in the email. High total -> spam. That's it. That is logistic regression — the most widely deployed model in the world.

Machine learning's contribution: you don't pick the points, the computer learns them from labelled examples.

How it actually works

Step 1 — Turn text into numbers (bag of words).

Build a vocabulary from all your emails, then count occurrences:

Vocabulary: [free, viagra, meeting, invoice, click]

"free free click here"        ->  [2, 0, 0, 0, 1]
"meeting about the invoice"   ->  [0, 0, 1, 1, 0]

Word order is thrown away — hence "bag" of words. Surprisingly, this works well for topic-level tasks.

Step 2 — Score it.

score = w1*count_free + w2*count_viagra + w3*count_meeting + ... + b

The ws are the learned points; b (the "bias") shifts the baseline.

Step 3 — Turn the score into a probability with the sigmoid.

A score can be anything from −∞ to +∞. Probability must be 0–1. The sigmoid function squashes it:

sigmoid(z) = 1 / (1 + e^(-z))

  z = -5  -> 0.007     (very confident: not spam)
  z =  0  -> 0.5       (no idea)
  z = +5  -> 0.993     (very confident: spam)

     1.0 |                        ______________
         |                   ____/
         |               ___/
     0.5 |............../..............
         |          ___/
         |     ____/
     0.0 |____/________________________________
         -6    -4   -2    0    2    4    6      <- score z

Step 4 — Learn the weights by the loop from Chapter 1: predict, measure error, nudge. (Chapter 10 explains the nudging.)

Worked example, by hand

Say the model has learned: w_free = 3, w_viagra = 8, w_meeting = -4, b = -2.

Email: "free free meeting" -> counts: free=2, viagra=0, meeting=1

score = 3*2 + 8*0 + (-4)*1 + (-2) = 6 + 0 - 4 - 2 = 0
sigmoid(0) = 0.5   ->  "I genuinely can't tell."

Email: "free viagra click" -> free=1, viagra=1

score = 3*1 + 8*1 - 2 = 9
sigmoid(9) = 0.9999  ->  spam, very confident

You can now read any logistic-regression model by eye. That's a real skill.

Practice (30 min) — build a working spam filter

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
import numpy as np

# A tiny toy dataset so you can see everything. (Real ones come next.)
emails = [
    "free money click here now", "win a free prize claim now",
    "cheap viagra buy now", "you have won free cash click",
    "limited offer free trial click here", "claim your free gift now",
    "meeting tomorrow at 10am", "please find the invoice attached",
    "can we reschedule our meeting", "the quarterly report is ready",
    "lunch on friday?", "here are the notes from the meeting",
]
labels = [1,1,1,1,1,1, 0,0,0,0,0,0]      # 1 = spam, 0 = not spam

vec = CountVectorizer()
X = vec.fit_transform(emails)            # the bag-of-words matrix
y = np.array(labels)

model = LogisticRegression()
model.fit(X, y)

# What did it learn? Print the "points" per word.
words = np.array(vec.get_feature_names_out())
weights = model.coef_[0]
order = np.argsort(weights)
print("Most HAM-ish words: ", list(zip(words[order][:5],  weights[order][:5].round(2))))
print("Most SPAM-ish words:", list(zip(words[order][-5:], weights[order][-5:].round(2))))

# Try it on new emails
tests = [
    "free click now",
    "can we move the meeting",
    "invoice attached please review",
    "claim your prize",
]
probs = model.predict_proba(vec.transform(tests))[:, 1]
for t, p in zip(tests, probs):
    print(f"  {p:.2f} spam   <-  {t!r}")

Now experiment:

Add an email "free lunch at the meeting" labelled 0. Retrain. How do the weights for "free" change?
Test with "FREE MONEY" (uppercase). Why does it fail? (Hint: look at CountVectorizer's lowercase setting — it defaults to True, so try CountVectorizer(lowercase=False) to see the failure.)
Test with "f r e e m o n e y". What does this tell you about how spammers attack such filters?

Scale it up (10 min)

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline

data = fetch_20newsgroups(subset="train",
        categories=["rec.sport.baseball", "sci.space"],
        remove=("headers","footers","quotes"))
test = fetch_20newsgroups(subset="test",
        categories=["rec.sport.baseball", "sci.space"],
        remove=("headers","footers","quotes"))

clf = make_pipeline(TfidfVectorizer(min_df=2), LogisticRegression(max_iter=1000))
clf.fit(data.data, data.target)
print(classification_report(test.target, clf.predict(test.data),
                            target_names=data.target_names))

You just built a text classifier with ~95% accuracy in six lines. This was a PhD topic in 1995.

Common confusions

"Regression" in the name? Historical accident. Logistic regression does classification. Ignore the name.
"Does it understand the email?" No. It counts words and adds points. It has no idea what "meeting" means. That limitation drives Parts 5 and 6.
Why is the sigmoid needed? To produce a probability, and because it makes the maths of learning work nicely.

Check yourself

What is a bag-of-words representation, and what does it throw away?
What does the sigmoid do and why do we need it?
If w_urgent = 5, is "urgent" evidence for or against spam?