Chapter 13 — How do you know if a model is any good?

The golden rule

Split your data BEFORE you do anything else:

   +-------------------- all your data --------------------+
   |  TRAIN (70-80%)      |  VALIDATION (10%) |  TEST (10%)|
   +----------------------+-------------------+------------+
      fit the knobs         choose settings     touch ONCE,
                            (learning rate,     at the very
                            model size, ...)    end

If you tune anything based on the test set, your test score becomes a lie.

Metrics: accuracy is usually the wrong one

Suppose 99% of emails are legitimate. A model that says "not spam" to everything gets 99% accuracy and is completely useless. You need:

                     PREDICTED
                  spam      not spam
            +-----------+-----------+
   spam     |    TP     |    FN     |   <- FN = spam that got through
ACTUAL      |  (caught) |  (missed) |
            +-----------+-----------+
  not spam  |    FP     |    TN     |   <- FP = real email in the spam folder
            | (false    |  (correct)|
            |  alarm)   |           |
            +-----------+-----------+

Precision = TP / (TP + FP)   "when I say spam, how often am I right?"
Recall    = TP / (TP + FN)   "of all the spam, how much did I catch?"
F1        = harmonic mean of precision and recall

Which one matters depends on the cost of each mistake:

Always ask "what is the cost of each kind of mistake?" before picking a metric.

Overfitting: the most common failure

error
  ^
  |  \                                       ___
  |   \                                 ____/     <- TEST error (what matters)
  |    \                          _____/
  |     \                    ____/
  |      \______________----/
  |       \___                                    <- TRAIN error
  |           \_____________________________
  +---------------------------------------------> training time / model size
      underfit      GOOD           overfit
                     ^
                     +-- stop here ("early stopping")

• Underfitting: bad on train AND test. Model too simple. Fix: bigger model, better features, train longer.
• Overfitting: great on train, bad on test. The model memorised instead of generalising. Fix: more data, simpler model, regularisation, early stopping.

Memorable analogy: underfitting is a student who didn't study. Overfitting is a student who memorised last year's exam answers word for word and fails when the questions change.

Data leakage: the sneaky killer

Leakage = information sneaking into training that won't exist at prediction time, or the test data influencing training. Symptom: suspiciously good scores that collapse in production.

Real examples:

• Predicting churn using a feature account_closed_date — which only exists because they churned.
• Scaling your features using statistics computed over the whole dataset (including the test set) before splitting.
• The same customer appearing in both train and test.
• Predicting stock prices with a random split instead of a time-based split.
• (LLM era) the benchmark's questions appearing in the pretraining data — called contamination, and a serious ongoing problem in AI evaluation.

A famous real case: a pneumonia detector scored 0.95 AUC in the lab and failed in the clinic. It had learned to recognise the portable X-ray machine marker — used on sicker, bed-bound patients — rather than the lungs. The fix was a hospital-stratified split.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report, confusion_matrix
import numpy as np

data = fetch_20newsgroups(subset="all",
        categories=["rec.sport.baseball", "sci.space", "talk.politics.misc"],
        remove=("headers","footers","quotes"))

X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, stratify=data.target, random_state=0)

vec = TfidfVectorizer(min_df=2)
Xtr = vec.fit_transform(X_train)     # fit ONLY on train  <-- avoids leakage
Xte = vec.transform(X_test)          # transform test with the SAME vocabulary

clf = LogisticRegression(max_iter=1000).fit(Xtr, y_train)
pred = clf.predict(Xte)

print(classification_report(y_test, pred, target_names=data.target_names))
print("Confusion matrix (rows=true, cols=predicted):")
print(confusion_matrix(y_test, pred))
print("\n5-fold CV on training data:",
      cross_val_score(clf, Xtr, y_train, cv=5).round(3))

# WRONG: fit the vectorizer on ALL data before splitting
vec_bad = TfidfVectorizer(min_df=2)
X_all = vec_bad.fit_transform(data.data)          # <-- sees the test set!
Xtr2, Xte2, ytr2, yte2 = train_test_split(X_all, data.target, test_size=0.2,
                                          stratify=data.target, random_state=0)
clf2 = LogisticRegression(max_iter=1000).fit(Xtr2, ytr2)
print("leaky score: ", clf2.score(Xte2, yte2))
print("honest score:", clf.score(Xte, y_test))