22. Statistics for ML & Experimentation

Level: Expert You'll use this for: trusting (or correctly distrusting) an A/B test, a model evaluation, or a paper's claimed result.

Maximum Likelihood Estimation (MLE) vs. MAP

MLE picks the parameter values that make the observed data most probable: θ_MLE = argmax P(data | θ). Training a model by minimizing cross-entropy loss is MLE under a specific probabilistic assumption about the output distribution — it's not a separate technique, it's the same math wearing a different name.

MAP (Maximum A Posteriori) adds a prior belief about θ: θ_MAP = argmax P(data|θ)·P(θ). Adding L2 regularization to a loss function is mathematically equivalent to assuming a Gaussian prior on the weights — regularization isn't an ad-hoc trick, it's Bayesian MAP estimation in disguise.

Hypothesis testing and p-values

A p-value answers: "if there were truly no effect (the null hypothesis), how likely would data this extreme (or more extreme) be?" A small p-value (conventionally < 0.05) is evidence against the null — it is not "the probability the null hypothesis is true," a near-universal misreading.

from scipy import stats

# A/B test: control conversion 100/1000, variant 130/1000
table = [
    [100, 900],   # control: converted, did not convert
    [130, 870],   # variant: converted, did not convert
]
chi2, p_value, dof, expected = stats.chi2_contingency(table)
print(p_value)   # if < 0.05, the observed gap is unlikely under "no difference"

For binary outcomes like conversion/no-conversion, use a proportion test or chi-square test like this. A t-test is more natural for continuous outcomes such as latency, revenue per user, or time spent.

Where the t-statistic itself comes from, computed by hand for two small samples (A = [10, 12, 9], B = [15, 14, 16]):

1. Mean of each group:
     mean_A = (10+12+9)/3 = 10.33
     mean_B = (15+14+16)/3 = 15.0

2. Standard deviation of each group (Ch. 9's method — deviations, square,
   average, sqrt):
     std_A ≈ 1.53      std_B ≈ 1.0

3. Standard error of the difference (roughly, for similar sample sizes):
     SE ≈ sqrt(std_A²/n_A + std_B²/n_B) = sqrt(1.53²/3 + 1.0²/3) ≈ 1.02

4. t-statistic = (difference in means) / (standard error):
     t = (15.0 - 10.33) / 1.02 ≈ 4.58

A t far from 0 (like 4.58) says "the gap between the two means is large relative to how much the data naturally jitters within each group" — the p_value is just that t translated into "how often would a gap this large happen by pure chance," via the t-distribution's known shape (the one calculation in this chapter genuinely worth leaving to a library, since it requires integrating a probability density).

Confidence intervals

A 95% confidence interval means: "if we repeated this experiment many times, 95% of the intervals we'd construct this way would contain the true value" — not "there's a 95% chance the true value is in this specific interval" (a frequentist/Bayesian distinction that trips up even experienced engineers).

p-hacking and multiple comparisons — the trap that invalidates most naive A/B testing

If you test 20 independent metrics at p < 0.05 significance, you expect ~1 false positive by pure chance alone — even with zero real effect.

import numpy as np
from scipy import stats

np.random.seed(0)
false_positives = 0
for _ in range(20):
    # Two random samples with NO real difference
    a = np.random.normal(0, 1, 1000)
    b = np.random.normal(0, 1, 1000)
    _, p = stats.ttest_ind(a, b)
    if p < 0.05:
        false_positives += 1
print(false_positives)   # expect ~1, purely from testing 20 things at once

This is the mathematical justification for Bonferroni correction (divide your significance threshold by the number of comparisons) and for being deeply skeptical of any "we tested 30 metrics and found 2 significant results" dashboard.

Cross-entropy loss and KL divergence, the ML-training connection

Cross-entropy(p, q) = -Σ p(x)·log(q(x))

where p is the true label distribution and q is the model's predicted distribution. Minimizing cross-entropy loss during classifier training is mathematically minimizing the KL divergence between the true and predicted distributions (Ch. 23 covers KL divergence directly) — this is why cross-entropy, not raw accuracy, is the loss function, even though accuracy is the metric you actually care about: it's differentiable and it directly optimizes distributional closeness.

MLE, derived with calculus: the coin and the bell

Bernoulli MLE. You observe k failures in n independent requests. What failure rate p makes that most likely?

likelihood:       ℒ(p) = pᵏ · (1-p)ⁿ⁻ᵏ
log-likelihood:   ℓ(p) = k·ln p + (n-k)·ln(1-p)          (logs turn products into sums, Ch. 5)
derivative:       ℓ'(p) = k/p - (n-k)/(1-p) = 0
solve:            k(1-p) = (n-k)p   ->   p̂ = k/n

The "obvious" estimate, failures ÷ total, is the maximum-likelihood estimate. Calculus confirms the intuition.

Gaussian MLE gives you MSE. Assume each target is the model's prediction plus Gaussian noise: yᵢ = f(xᵢ; θ) + noise, with noise ~ N(0, σ²). The log-likelihood is:

ℓ(θ) = Σᵢ ln[ 1/(σ√(2π)) · e^(-(yᵢ - f(xᵢ;θ))² / (2σ²)) ]
     = constant  -  (1/(2σ²)) · Σᵢ (yᵢ - f(xᵢ; θ))²

Maximizing ℓ is the same as minimizing the sum of squared errors. MSE loss, least squares (Ch. 10) and linear regression are all MLE under a Gaussian-noise assumption. Change the noise assumption and the loss changes with it: Laplace noise gives absolute error (MAE), and Bernoulli outputs give cross-entropy. Choosing a loss function is choosing a probability model.

The two-proportion z-test, by hand

Here's the A/B test above (control 100/1000, variant 130/1000) done by hand. Under the null hypothesis "no difference", both groups share one pooled rate:

p̂_pool = (100 + 130) / (1000 + 1000) = 0.115

SE = √( p̂_pool · (1 - p̂_pool) · (1/n₁ + 1/n₂) )
   = √( 0.115 · 0.885 · 0.002 ) = √0.0002036 ≈ 0.01427

z = (p̂₂ - p̂₁) / SE = (0.13 - 0.10) / 0.01427 ≈ 2.10

p-value = 2 · P(Z > 2.10) ≈ 0.035

scipy's chi2_contingency in the code above prints about 0.042, not 0.035, because it applies Yates' continuity correction by default. Pass correction=False and it matches the hand calculation exactly. A 2×2 chi-square test is this z-test, squared (χ² = z²).

Confidence intervals, the formula

p̂ ± z_(α/2) · √( p̂(1 - p̂) / n )

Example: variant at 130/1000: 0.13 ± 1.96 · √(0.13 · 0.87 / 1000) = 0.13 ± 0.021 = [10.9%, 15.1%]. The √n in the denominator is the CLT's law of diminishing returns again: to halve the interval's width you need 4× the traffic.

How many users does an A/B test need? Power analysis

Before you launch, decide the smallest effect worth detecting, then solve for n:

n per group = (z_(α/2) + z_β)² · [ p₁(1-p₁) + p₂(1-p₂) ] / δ²

n = (1.96 + 0.84)² · (0.10·0.90 + 0.11·0.89) / 0.01²
  = 7.85 · 0.1879 / 0.0001  ≈  14,750 users per group

Detecting a 1-point lift on a 10% baseline takes about 30,000 users in total. Because of δ², detecting half that lift takes 4× as many. This is why small companies can't run the same experiments as large ones, and why "we ran it for a day and it looked good" usually means "underpowered."

Networking in the wild: the rule of three

You send 3,000 probe packets across a link and none are lost. Is the loss rate zero? No, but you can bound it. The 95% upper bound p is the rate at which seeing zero losses still has a 5% chance:

(1 - p)ⁿ = 0.05   ->   n·ln(1 - p) = ln 0.05   ->   n·(-p) ≈ -3.0   ->   p ≈ 3/n

(using ln(1-p) ≈ -p from Ch. 12's Taylor expansion, and -ln 0.05 ≈ 3.0)

Zero failures in n trials means the true rate is below 3/n with 95% confidence. For 3,000 clean probes, loss is below 0.1%. If your SLO is 0.01% loss, 3,000 probes can't verify it. You need 30,000. The same rule answers "we ran 500 red-team prompts and the model never leaked the system prompt". That only shows the leak rate is below 3/500 = 0.6%, which is not zero.

Real-life engineering ties

• AI: correctly interpreting model evaluation metrics, understanding why regularization works probabilistically, designing experiments that won't be invalidated by multiple-comparisons issues.
• SWE/Product: every feature-flag A/B test at a company runs on this math — misreading a p-value here directly costs the business money (shipping "winning" features that were actually noise).
• Security: evaluating a new detection rule/model requires the same rigor — "we tested it and false positives dropped" needs a confidence interval, not just a single before/after number, especially against naturally noisy attack-volume data.

More real-life examples:

• The 1973 UC Berkeley graduate admissions data is a famous, documented real-world case of Simpson's Paradox — the university appeared to favor male applicants overall, but broken down by department, most departments individually favored female applicants; the aggregate trend reversed once properly segmented.
• The ongoing "replication crisis" in psychology and biomedical research (many high-profile 2010s replication attempts failing) is widely attributed in part to the multiple-comparisons and p-hacking problems demonstrated in this chapter's code example.
• Large tech companies with heavy experimentation cultures (Booking.com, Airbnb, Microsoft's ExP platform) run thousands of simultaneous A/B tests and invest in statistical infrastructure (sequential testing, false-discovery-rate control) to guard against inflated false-positive rates.
• Clinical drug trials conventionally require p < 0.05 for a single pre-registered primary endpoint specifically because pre-registration prevents the multiple-comparisons/p-hacking trap — a regulatory rule that is a direct, high-stakes application of this chapter's math.

Common mistake

Peeking — checking A/B test results repeatedly and stopping "as soon as it's significant." This inflates the false-positive rate far above the nominal 5%, because you're implicitly running many sequential tests (similar to the multiple-comparisons problem above). Fix: pre-register a sample size/stopping rule, or use sequential testing methods designed for this (e.g., always-valid p-values).