16. Bayes' Theorem

Level: Intermediate You'll use this for: spam filters, medical/security test interpretation, and understanding why "99% accurate" detectors can still be mostly wrong.

P(A|B) = [ P(B|A) · P(A) ] / P(B)

Applying the formula by hand, with natural frequencies (no algebra needed)

The cleanest way to compute Bayes' theorem by hand is to skip the fraction and just count people. Say 1% of a population of 1,000 has a disease, and a test is 90% accurate both ways:

1,000 people total
  -> 10 actually sick   (1% of 1,000)
  -> 990 actually healthy

Of the 10 sick people, the test catches 90%:
  -> 9 correctly test positive (true positives)
  -> 1 sick person tests negative (missed)

Of the 990 healthy people, the test correctly clears 90%:
  -> 891 correctly test negative
  -> 99 healthy people wrongly test positive (false positives)

Total positive tests = 9 (true) + 99 (false) = 108

P(actually sick | tested positive) = 9 / 108 ≈ 0.083   -> only ~8.3%!

That fraction — true positives divided by all positives — is Bayes' theorem with the algebra hidden: you never had to touch P(B|A)·P(A)/P(B) directly, just count outcomes in a hypothetical group of real people. This "natural frequency" method is the fastest way to sanity-check any Bayes-flavored claim without a calculator.

The base rate fallacy — the single most important lesson in this chapter

Suppose an intrusion detection system is 99% accurate (both true-positive and true-negative rate), and genuinely malicious events are rare: 1 in 10,000 events. Out of 1,000,000 events/day, how many alerts are actually malicious?

p_malicious = 1 / 10_000
p_benign = 1 - p_malicious
accuracy = 0.99

total_events = 1_000_000
actual_malicious = total_events * p_malicious        # 100
actual_benign    = total_events * p_benign            # 999,900

true_positives  = actual_malicious * accuracy         # 99
false_positives = actual_benign * (1 - accuracy)      # 9,999

precision = true_positives / (true_positives + false_positives)
print(precision)   # ≈ 0.0098 -> under 1% of alerts are real!

This is not a contrived example — it is the mathematical reason "alert fatigue" is a structural, unavoidable problem for any detector applied to a rare-event population, no matter how "accurate" the detector sounds. When the base rate of the thing you're looking for is very low, even a low false-positive rate produces a false-positive majority in absolute terms. This is a direct, load-bearing consequence of Bayes' theorem and must inform how any Sr. security/ML engineer reads a confusion matrix (Ch. 22 and 26 build directly on this with precision/recall/ROC curves).

Naive Bayes classifier (spam filtering)

Bayes' theorem, applied to classification: given the words in an email, what's P(spam | words)? "Naive" because it assumes words are conditionally independent given the class (usually false, but works well in practice).

P(spam | word1, word2, ...) ∝ P(spam) · P(word1|spam) · P(word2|spam) · ...

Multiplying hundreds of small probabilities underflows to 0.0 in floating point (Ch. 1), so real implementations add log-probabilities instead (Ch. 5): log P(spam) + Σ log P(wordᵢ | spam). The ∝ ("proportional to") means we dropped the denominator P(words). It's the same for both classes, so it can't change which one wins.

Every symbol in Bayes' theorem has a name

              likelihood   prior
              ┌───┴───┐   ┌─┴─┐
P(H | E)  =   P(E | H)  · P(H)
└──┬───┘      ─────────────────
posterior          P(E)
                   └─┬─┘
                  evidence

The formula is "posterior ∝ likelihood × prior". Every learning system that updates beliefs from data, from spam filters to Kalman filters in GPS to Bayesian optimization of hyperparameters, runs this multiplication.

Bayesian updating in practice: the Beta distribution

How do you estimate a rate (a conversion rate, a link's loss rate, a model's accuracy) and how uncertain you are about it? Use a Beta distribution as the belief about the unknown rate θ:

prior:      θ ~ Beta(α, β)
observe:    k successes in n trials
posterior:  θ ~ Beta(α + k, β + n - k)          mean = (α + k) / (α + β + n)

This is called a conjugate prior: Beta prior × binomial likelihood = Beta posterior. A Bayesian update is two additions.

Example: a new feature variant, flat prior Beta(1, 1). On day 1 you see 13 conversions in 100 visitors, so the posterior is Beta(14, 88), with mean 14/102 ≈ 13.7% and a 95% credible interval of about [7.8%, 21.0%]. The interval is wide because 100 visitors isn't much. After 10,000 visitors it would narrow to about ±0.7 points.

AI in the wild: Thompson sampling. To choose among several variants (ads, recommendation slots, LLM prompts), draw one random sample from each variant's Beta posterior and show the one with the highest draw. Uncertain variants sometimes produce high draws, so they get explored. Proven variants usually win, so they get exploited. This "multi-armed bandit" strategy wastes far less traffic than a fixed 50/50 A/B test, and it runs in production recommendation and ad systems.

Networking in the wild: the same update tracks a link's delivery probability from probe results (each probe is a Bernoulli trial), with a built-in uncertainty that shrinks as probes accumulate. That's more honest than an EWMA (Ch. 9) when probes are sparse.

from scipy.stats import beta
post = beta(1 + 13, 1 + 87)            # flat prior + 13 of 100
print(post.mean(), post.interval(0.95))  # 0.137, (0.078, 0.210)

import numpy as np                       # Thompson sampling: pick the arm with the highest draw
rng = np.random.default_rng(0)
arms = {"A": (1 + 120, 1 + 880), "B": (1 + 13, 1 + 87)}   # (α, β) per variant
draws = {k: rng.beta(a, b) for k, (a, b) in arms.items()}
print(max(draws, key=draws.get), draws)

Real-life engineering ties

• SWE: interpreting any classifier's output correctly instead of treating "confidence score" as "probability of being right."
• AI: Naive Bayes classifiers, Bayesian A/B testing, Bayesian hyperparameter optimization; Bayes is also the formal foundation of updating a model's belief as new data arrives (Bayesian inference, Ch. 22).
• Network: anomaly detection systems that flag "unusual" traffic must be evaluated with base-rate awareness — most traffic is normal, so even a "good" anomaly score threshold floods you with false positives.
• Security: this is arguably the single highest-leverage math concept for a security engineer — every SIEM tuning decision, every "should we lower this detection threshold" conversation, is implicitly a Bayes' theorem question about precision vs. recall tradeoffs.

More real-life examples:

• The classic medical teaching example — a mammogram with 90% sensitivity screening for a cancer with 1% prevalence — produces a majority of positive results being false positives, for exactly the same base-rate reason as the IDS example above; doctors are famously shown to get this calculation wrong as often as patients.
• Early spam filters (SpamAssassin, and Paul Graham's influential 2002 "A Plan for Spam" essay) popularized Naive Bayes spam classification, directly kicking off widespread industry adoption of Bayesian filtering.
• Payment fraud systems must explicitly balance false declines (blocking a legitimate purchase) against missed fraud, and because legitimate transactions vastly outnumber fraudulent ones, a fraud model's naive "accuracy" is nearly meaningless — precision/recall framed around the true (low) fraud base rate is what actually matters.
• Recommendation systems handle new users with no history (the "cold start" problem) by falling back to a prior — a population-average preference distribution — and updating toward personalized recommendations as interaction data arrives, a live, continuous application of Bayesian updating.

Common mistake

Reading "99% accurate" and concluding "99% of alerts are real." Accuracy (or even true-positive rate) says nothing about precision without knowing the base rate — always ask "how rare is the thing being detected" before trusting a detector's headline number.