8. Basic Probability

Level: Beginner You'll use this for: reasoning about A/B tests, retry logic, and every "how likely is this" question, including security alert quality.

Core definitions

• Sample space (Ω): every possible outcome.
• Event (A): a subset of outcomes you care about.
• P(A): a number from 0 to 1. P(A) = 0 never happens, P(A) = 1 always happens.

Independence vs. dependence

Two events are independent if one doesn't affect the other's probability: P(A ∩ B) = P(A) · P(B). Coin flips are independent (the coin has no memory) — this is why "I've flipped 5 heads in a row, tails is due" (the gambler's fallacy) is simply wrong. Each flip is still 50/50.

P(fail once) = 1 - p = 0.3
P(fail 3 times in a row) = 0.3³ = 0.027   (2.7%)
P(at least one success in 3 tries) = 1 - 0.027 = 0.973

p_fail = 1 - 0.7
p_all_fail = p_fail ** 3
print(1 - p_all_fail)   # 0.973

Union of events (why "or" isn't just addition)

P(A ∪ B) = P(A) + P(B) - P(A ∩ B)

You subtract the overlap so you don't double count it. Forgetting the -P(A ∩ B) term is the single most common basic probability bug.

The three axioms everything else is built on

In 1933 Andrey Kolmogorov showed that all of probability theory follows from three rules. Every formula in this guide that involves P(...), from Bayes to the Poisson distribution, can be derived from these three.

1.  P(A) ≥ 0                                  for every event A
2.  P(Ω) = 1                                  something in the sample space happens
3.  P(A ∪ B) = P(A) + P(B)                    when A and B can't both happen (A ∩ B = ∅)

Example: the complement rule, P(not A) = 1 - P(A), isn't a separate fact you memorize. A and not A can't both happen, and together they cover Ω. So by axiom 3, P(A) + P(not A) = P(Ω), and by axiom 2 that equals 1. The retry example above (1 - 0.027) used exactly this.

Conditional probability: updating on new information

P(A | B) = P(A ∩ B) / P(B)          (defined when P(B) > 0)

Read it as zooming in. Once you know B happened, every outcome outside B is gone. B becomes the new universe, and you ask what fraction of it also lies inside A.

Example: from logs, 2% of requests are to /checkout, and 0.5% of all requests are /checkout requests that failed. What is the failure rate of checkout requests?

P(fail | checkout) = P(fail ∩ checkout) / P(checkout) = 0.005 / 0.02 = 0.25  -> 25%

A global failure rate that looks fine can hide one endpoint that fails a quarter of the time. This is why SLO dashboards slice by route.

The law of total probability: divide and conquer

If B₁, B₂, …, Bₙ split the world into non-overlapping cases that cover everything:

P(A) = Σᵢ P(A | Bᵢ) · P(Bᵢ)

Example (networking): a global load balancer sends 50% of traffic to zone 1, 30% to zone 2 and 20% to zone 3. The per-zone packet-loss rates are 0.1%, 0.2% and 1%. What is the overall loss rate?

P(loss) = 0.001·0.5 + 0.002·0.3 + 0.010·0.2
        = 0.0005 + 0.0006 + 0.0020
        = 0.0031   -> 0.31%

Now run it backwards with Bayes (Ch. 16): given that a packet was lost, how likely is it that it went through zone 3? 0.0020 / 0.0031 ≈ 65%. Zone 3 carries only 20% of traffic but causes 65% of the loss, so that's where to look first. Total probability plus Bayes is the maths behind "which component is responsible for most of the errors?"

Random variables and expected value

A random variable X is a number whose value is decided by chance, such as the number of retries, a request's latency, or the bytes in a flow. Its expected value is the long-run average you'd see over many repetitions:

E[X] = Σₓ x · P(X = x)                 (discrete: weighted sum of outcomes)
E[X] = ∫ x · f(x) dx                    (continuous: weighted integral, Ch. 12)

Example: how many attempts does a flaky call take, on average? Each attempt succeeds independently with probability p. The number of attempts until the first success follows a geometric distribution, and its expectation works out to:

E[attempts] = 1·p + 2·(1-p)·p + 3·(1-p)²·p + ...  =  1/p

With p = 0.7, that's 1/0.7 ≈ 1.43 attempts per call on average, so a retrying client puts about 43% more load on the backend than a non-retrying one. A backend that fails more (smaller p) gets more traffic. That feedback loop is how retry storms happen.

Networking in the wild: the ETX routing metric. In wireless mesh networks (MIT's Roofnet, then the Linux OLSR and B.A.T.M.A.N. routing daemons), a link's cost is its expected transmission count. If a data frame gets through with probability d_f and its ACK comes back with probability d_r, an attempt succeeds with probability d_f · d_r, so:

ETX = 1 / (d_f · d_r)

Hop-count routing picks the 1-hop path because it has fewer hops. ETX routing picks the 2-hop path because, on average, it uses the air about 40% less. The difference is just expected value.

Linearity of expectation: the most useful "free lunch" in probability

E[X + Y] = E[X] + E[Y]         ALWAYS, even if X and Y are dependent
E[c · X] = c · E[X]            for any constant c

The surprising part is that this works without independence. That lets you break a hard count into easy indicator pieces and add up their expectations.

Example: expected hash collisions. Insert n = 1000 keys into a hash table with m = 10,000 buckets. How many pairs of keys share a bucket, on average? For each of the C(n,2) pairs (Ch. 15), let an indicator be 1 if that pair collides. Each pair collides with probability 1/m. So:

E[colliding pairs] = C(n,2) · (1/m) = (1000·999/2) / 10000 = 499,500 / 10,000 ≈ 50

That's about 50 collisions with the table only 10% full. Linearity of expectation answered it in one line, with no need to work out the messy joint distribution. The same argument gives the birthday bound in Ch. 24.

AI in the wild: why dropout rescales. During training, dropout zeroes each neuron's output h with probability 1 - q (so it keeps it with probability q). The expected output becomes E[kept h] = q·h + (1-q)·0 = q·h. At inference time nothing is dropped, so the next layer would suddenly see inputs 1/q times larger than it was trained on. "Inverted dropout", the default in PyTorch, divides by q during training. By linearity, E[h/q · keep] = (1/q)·q·h = h, so the expected value is the same in training and inference.

Variance of a random variable

Var(X) = E[(X - E[X])²] = E[X²] - (E[X])²

Example: a single request that fails with probability p (a Bernoulli variable, 1 = fail, 0 = success). E[X] = p, and since X² = X for 0/1 values, E[X²] = p. So Var(X) = p - p² = p(1-p). This is largest at p = 0.5 and tiny near 0 or 1. That's why a rare-error metric (p = 0.001) needs huge sample sizes before you can trust a change in it, which Ch. 22 turns into a sample-size formula.

Real-life engineering ties

• SWE: retry/circuit-breaker design, error budget math (SRE), load test result interpretation.
• AI: every classifier outputs a probability; understanding calibration (does "90% confident" actually mean right 90% of the time?) starts here.
• Network: packet loss modeling, redundant path availability (1 - (1-p)^n for n independent paths — Ch. 25 combines this with queueing).
• Security: false positive rate of a detection rule is a probability; understanding independence vs. correlation of signals is the difference between a SIEM rule that actually reduces risk and one that just generates alert fatigue (see Bayes, Ch. 16).

More real-life examples:

• Cloud architects use independent-failure probability math directly when deciding how many Availability Zones to span: P(all zones down) = p1 · p2 · p3 only holds if the zones truly fail independently — the same assumption checked in the common mistake below.
• Circuit-breaker libraries (Netflix Hystrix, resilience4j) are built on the same "probability of success across independent retries" reasoning as the retry example above, and track a rolling failure probability to decide when to "open" the breaker.
• CAPTCHA and bot-detection systems reason about the probability that a sequence of behaviors (mouse movement, timing, request pattern) is exhibited by a human vs. a script — a compound probability judgment, not a single yes/no rule.
• Credential-stuffing attacks exploit the (very non-independent) fact that many users reuse the same password across services — attackers rely on this correlation, not on independent per-site guessing, which is exactly why password reuse defeats per-site rate-limiting defenses.

Common mistake

Assuming events are independent when they're not (e.g., "each server has a 1% chance of failing, so with 100 servers only 1 fails" ignores correlated failures — a single power/network outage can take down all 100 at once). This is the mathematical root of "the cloud availability zone problem."