56. scikit-learn: the fit/predict contract, pipelines, and leakage

56.1 The contract

Every scikit-learn estimator follows one interface: fit(X, y) learns from data, predict(X) (or transform(X)) applies what was learned. Learn it once, and you can swap a logistic regression for a random forest by changing one line.

56.2 Real-world example: the help-desk ticket classifier

The AI guide's running example is routing help-desk tickets. Here it is as a real pipeline:

# labs/tickets.py
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(42)
templates = {
    "billing":   ["refund for order {n}", "charged twice on invoice {n}", "update my credit card",
                  "wrong amount on bill {n}", "cancel subscription and refund"],
    "technical": ["app crashes on login", "error 500 when saving", "cannot reset password",
                  "page loads forever on {n}", "sync fails with timeout error"],
    "shipping":  ["package {n} not delivered", "tracking shows no update", "delivered to wrong address",
                  "order {n} arrived damaged", "when will my order ship"],
}
noise = ["please help", "urgent", "thanks", "hi team", "asap", ""]
texts, labels = [], []
for label, ts in templates.items():
    for _ in range(200):
        t = rng.choice(ts).format(n=rng.integers(1000, 9999))
        texts.append(f"{rng.choice(noise)} {t} {rng.choice(noise)}".strip())
        labels.append(label)

X_train, X_test, y_train, y_test = train_test_split(
    texts, labels, test_size=0.2, stratify=labels, random_state=0)

model = make_pipeline(
    TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True),
    LogisticRegression(max_iter=1000),
)
print("5-fold CV accuracy:", cross_val_score(model, X_train, y_train, cv=5).mean().round(3))
model.fit(X_train, y_train)
pred = model.predict(X_test)
print(classification_report(y_test, pred, digits=3))
print(confusion_matrix(y_test, pred, labels=list(templates)))

for t in ["I was charged twice, need my money back", "the app shows an error after update",
          "my parcel never arrived"]:
    probs = model.predict_proba([t])[0]
    print(f"{model.classes_[probs.argmax()]:<10} {probs.max():.2f}  {t}")

vec, clf = model.named_steps["tfidfvectorizer"], model.named_steps["logisticregression"]
terms = np.array(vec.get_feature_names_out())
for i, cls in enumerate(clf.classes_):
    print(cls, "<-", ", ".join(terms[np.argsort(clf.coef_[i])[-5:]]))   # explainability for free

This is the AI guide's first model and TF-IDF as working code. A TF-IDF + logistic-regression baseline trains in milliseconds, is explainable (the last loop prints the words that drive each class), and is surprisingly hard to beat on short, domain-specific text. Always build this baseline before reaching for an LLM.

56.3 Leakage: the bug that makes models look great

# WRONG: the vectorizer sees the test set's vocabulary and document frequencies
# vec = TfidfVectorizer().fit(all_texts); X = vec.transform(all_texts); then split X

Any statistic computed on the full dataset before splitting (vocabulary, mean and standard deviation for scaling, imputation values, feature selection) leaks information from the test set into training. Your offline accuracy goes up, and production accuracy does not. The fix is the pipeline: put every preprocessing step inside it, so fit only ever sees training data, including inside each cross-validation fold. The AI guide Ch 13 covers evaluation in depth. Other leaks to watch for: duplicates across train and test (Ch 60 deduplicates), random splits on time-series data (split by time instead), and features that are only known after the label (a "refund_issued" column in a churn model).

56.4 Saving the model

import joblib
# joblib.dump(model, "tickets.joblib")      # a pickle: load only your own files (Ch 47.1)

For models that leave your machine, prefer an export format: skops for scikit-learn with a safer loader, or ONNX (skl2onnx) to run the model from Go, Rust, or C++ with ONNX Runtime, which fits the "Python offline, another language online" split from §0.5.