12. Calculus I — Limits, Derivatives, Rates of Change

Level: Intermediate You'll use this for: understanding "rate of change" metrics, and as the prerequisite for all of ML optimization.

f'(x) = lim(h→0) [f(x+h) - f(x)] / h

Watching that limit converge, numerically

For f(x) = x² at x = 3, plug in shrinking values of h and watch [f(x+h) - f(x)] / h settle down:

h = 1:      [f(3+1) - f(3)] / 1     = [16 - 9] / 1     = 7.0
h = 0.1:    [f(3.1) - f(3)] / 0.1   = [9.61 - 9] / 0.1  = 6.1
h = 0.01:   [f(3.01) - f(3)] / 0.01 = [9.0601-9]/0.01  = 6.01
h = 0.001:  ...                                          = 6.001

As h -> 0, the ratio converges to 6.

The power rule says f'(x) = 2x, so f'(3) = 2×3 = 6 — matching exactly what the shrinking-h calculation converges to. That's what the "limit" in the definition means in practice: it's not a mysterious symbol, it's "keep shrinking h and see what number the ratio approaches."

Common derivative rules

d/dx (x^n) = n·x^(n-1)          power rule
d/dx (e^x) = e^x                 e^x is its own derivative
d/dx (ln x) = 1/x
d/dx (f(x)+g(x)) = f'(x) + g'(x) sum rule
d/dx (f(x)·g(x)) = f'(x)g(x) + f(x)g'(x)   product rule

import sympy as sp
n = sp.symbols('n')
cost = 0.01*n**2 + 5*n
marginal = sp.diff(cost, n)
print(marginal)             # 0.02*n + 5
print(marginal.subs(n, 100))  # marginal cost at n=100 requests/sec

The derivative, symbol by symbol

f'(x) = lim(h→0) [f(x+h) - f(x)] / h        also written  df/dx,  d/dx f(x)

Units check: if f is bytes and x is seconds, f' is bytes/second. A derivative always has units of output per input.

Deriving a rule instead of memorizing it

Here's the power rule for f(x) = x², straight from the definition:

[f(x+h) - f(x)] / h = [(x+h)² - x²] / h
                    = [x² + 2xh + h² - x²] / h
                    = [2xh + h²] / h
                    = 2x + h          ->  as h → 0:   f'(x) = 2x

The h² term was "too small to matter", and that's the entire spirit of calculus: zoom in far enough and every smooth curve looks like a straight line.

Two more rules you'll need, plus the chain rule (Ch. 13 develops it fully):

d/dx [f(x)/g(x)] = [f'(x)·g(x) - f(x)·g'(x)] / g(x)²     quotient rule
d/dx f(g(x))     = f'(g(x)) · g'(x)                        chain rule

AI in the wild: the sigmoid's derivative and vanishing gradients

The sigmoid squashes any number into a probability (0, 1). It's the output of every logistic regression and every binary classifier:

σ(x) = 1 / (1 + e^(-x))

Differentiate with the chain rule. The outer function is u⁻¹, the inner is 1 + e^(-x):

σ'(x) = -1·(1 + e^(-x))⁻² · (-e^(-x))
      = e^(-x) / (1 + e^(-x))²
      = [1/(1 + e^(-x))] · [e^(-x)/(1 + e^(-x))]
      = σ(x) · (1 - σ(x))

The derivative is built from the function's own output, so backprop gets it almost for free. It also has a dark side:

Each sigmoid layer multiplies the gradient by at most 0.25. Through 10 layers that's 0.25¹⁰ ≈ 0.000001, so the early layers barely learn. This one-line derivative is the reason deep networks switched to ReLU, whose derivative is exactly 1 for positive inputs (Ch. 21).

Taylor's theorem: every smooth function is secretly a polynomial

Theorem (Taylor). Near a point a, a smooth function equals its derivatives arranged as a polynomial:

f(x) = f(a) + f'(a)(x-a) + f''(a)/2! · (x-a)² + f'''(a)/3! · (x-a)³ + ...

Keep only the first two terms and you get the linear approximation f(x) ≈ f(a) + f'(a)(x-a). That's the "zoom in until it's a line" idea as a formula.

Examples you've already relied on without knowing it:

e^x       ≈ 1 + x + x²/2 + x³/6      at x = 0.1:  1.1051667  (true: 1.1051709)
ln(1 + x) ≈ x                         at x = 0.01: 0.01       (true: 0.00995)

The second one derives Ch. 6's Rule of 72. Money growing at rate r doubles when (1+r)ᵗ = 2, so t = ln 2 / ln(1+r) ≈ 0.693 / r. That gives "69.3 divided by the percentage rate", and 72 is used instead because it has more divisors.

Where Taylor shows up in AI: gradient descent (Ch. 21) is the first-order Taylor approximation. It assumes the loss is locally a plane and steps downhill on it. Newton's method and second-order optimizers keep the f'' term and model the loss as a bowl. The GELU activation used in GPT and BERT ships with a tanh-based approximation that's tuned to match the true curve closely and runs faster on GPUs.

Newton's method: square roots in four steps

To solve f(x) = 0, start with a guess, take the tangent line (the linear approximation above), and jump to where it hits zero:

x_{n+1} = x_n - f(x_n) / f'(x_n)

Example: compute √2, which is the root of f(x) = x² - 2 with f'(x) = 2x. The update simplifies to x_{n+1} = (x_n + 2/x_n) / 2:

x₀ = 1
x₁ = (1 + 2/1) / 2           = 1.5
x₂ = (1.5 + 2/1.5) / 2       = 1.41666...
x₃ = (1.41667 + 2/1.41667)/2 = 1.4142157
x₄                           = 1.41421356237469    (true: 1.41421356237310)

The number of correct digits roughly doubles every step (1, 3, 6, 12). That's called quadratic convergence. Your CPU's square-root and division units, the famous Quake III "fast inverse square root" hack, and the implied-volatility solvers in finance all finish with Newton steps.

Optimization: where the derivative is zero

At a smooth minimum or maximum the tangent is flat, so f'(x) = 0. The second derivative tells you which one you found:

f'(x*) = 0  and  f''(x*) > 0   ->  local minimum  (curves up, like a bowl)
f'(x*) = 0  and  f''(x*) < 0   ->  local maximum  (curves down, like a hill)

Example: right-sizing a fleet. Each server costs $2/hour, and the latency penalty to the business falls as you add servers, about 800/n dollars/hour:

C(n)   = 2n + 800/n
C'(n)  = 2 - 800/n²   = 0   ->   n² = 400   ->   n* = 20 servers
C''(n) = 1600/n³ > 0                          ->   minimum ✓

C(10) = $100/h     C(20) = $80/h     C(40) = $100/h

At the optimum, the marginal cost of one more server equals the marginal saving from it (2 = 800/n²). Every "find the sweet spot" engineering trade-off has this shape: cache size vs. hit rate, batch size vs. latency, replication factor vs. durability.

Networking in the wild: how sensitive is TCP to packet loss?

The Mathis formula approximates steady-state throughput for a loss-based TCP (Reno-style) flow:

throughput ≈ (MSS / RTT) · (C / √p)

throughput = (1460·8 bits / 0.1 s) · (1.22 / √0.0001)
           = 116,800 · 122.5  ≈ 14.3 Mbit/s

Now ask the calculus question: how sensitive is throughput to loss?

d(throughput)/dp = -½ · throughput / p

so a 1% relative increase in loss costs about 0.5% of throughput. Doubling the loss rate divides throughput by √2 (−29%, down to 10.1 Mbit/s). Going to 1% loss cuts it 10× to 1.4 Mbit/s. And because RTT sits in the denominator, the same loss rate hurts a transatlantic flow 10× more than a metro one. These two derivatives are why loss-based congestion control struggles on long, slightly lossy paths, and why Google built BBR, which models bandwidth and RTT directly instead of reacting to loss.

Integrals: the accumulated total

If the derivative answers "how fast right now?", the integral answers "how much in total?". It adds up a rate over time, or a density over a range, into an amount.

∫ₐᵇ f(x) dx  =  lim(n→∞) Σᵢ₌₁ⁿ f(xᵢ) · Δx        where Δx = (b - a)/n

Units check: if f is bytes/second and x is seconds, the integral is bytes. Integration multiplies units by the units of dx, and differentiation divides them.

Watching the rectangles converge, for ∫₀³ x² dx using midpoint rectangles:

n = 3 rectangles:    8.75
n = 6:               8.9375
n = 30:              8.9975
n = 300:             8.999975       ->  converges to exactly 9

Same idea as the derivative's shrinking h: the "limit" just means "keep slicing thinner and see what the sum approaches."

The Fundamental Theorem of Calculus

The theorem that welded derivatives and integrals into one subject (Newton and Leibniz, 1660s–1680s):

Theorem (FTC). If F' = f (that is, F is an antiderivative of f), then

∫ₐᵇ f(x) dx = F(b) - F(a)

And in the other direction: d/dx ∫ₐˣ f(t) dt = f(x). Accumulating a rate, then asking how fast the total is growing, gives you back the rate.

Integration and differentiation are inverses, so every derivative rule read backwards is an integral rule:

∫ xⁿ dx   = xⁿ⁺¹/(n+1) + C      (n ≠ -1)
∫ eˣ dx   = eˣ + C
∫ 1/x dx  = ln|x| + C
∫ e^(-λt) dt = -e^(-λt)/λ + C

(C is the "constant of integration". Constants have zero derivative, so an antiderivative is only fixed up to a constant, and it cancels in F(b) - F(a).)

Example: ∫₀³ x² dx = [x³/3]₀³ = 27/3 - 0 = 9. That's the 9 the rectangles were crawling toward, found exactly in one line.

Networking in the wild: rates, totals, and Prometheus

Your monitoring stack is a calculus engine:

Example 1, exact: a transfer ramps up linearly, r(t) = 2t Gbit/s, for 10 seconds. Total data:

∫₀¹⁰ 2t dt = [t²]₀¹⁰ = 100 Gbit = 12.5 GB

Its average rate is 10 Gbit/s, half the 20 Gbit/s peak. If you capacity-plan a link off the average, you'll saturate it at the end of every ramp.

Example 2, from samples (the trapezoid rule): you only have scraped rates, one every 15 s: 100, 140, 180, 160, 120 Mbit/s. Approximate each 15 s slice as a trapezoid (average of its two ends × width):

total ≈ 15 · [ (100+140)/2 + (140+180)/2 + (180+160)/2 + (160+120)/2 ]
      = 15 · [ 120 + 160 + 170 + 140 ]
      = 8,850 Mbit  ≈  1,106 MB  in one minute

That's how volume-based billing computes "total GB transferred" from sampled rates. (Transit billed on the 95th percentile uses Ch. 9's percentiles instead.) A token-bucket rate limiter is the same integral run live: the bucket holds ∫ refill_rate dt - consumed, capped at the bucket size.

AI in the wild: probability is integration

For a continuous random variable with density f(x) (Ch. 14), probabilities are areas:

P(a ≤ X ≤ b) = ∫ₐᵇ f(x) dx            ∫₋∞^∞ f(x) dx = 1            E[X] = ∫ x·f(x) dx

Example: the exponential distribution (time to next failure, rate λ):

f(t) = λ·e^(-λt)        for t ≥ 0

Total probability:  ∫₀^∞ λe^(-λt) dt = [-e^(-λt)]₀^∞ = 0 - (-1) = 1   ✓

P(failure within T) = ∫₀ᵀ λe^(-λt) dt = 1 - e^(-λT)

With MTBF 720 hours (λ = 1/720) and T = 100 h: 1 - e^(-100/720) = 1 - e^(-0.139) ≈ 13%. That's exactly what expon.cdf in Ch. 14 returns, because the CDF is this integral.

The Gaussian integral, a theorem that explains a constant:

∫₋∞^∞ e^(-x²) dx = √π

(The proof squares the integral, switches to polar coordinates, and the π falls out of the circle. It's one of the most elegant tricks in mathematics.) This is why the normal distribution's formula carries that odd 1/√(2π): it's the constant needed to make the total area exactly 1.

ROC AUC is literally an integral. The "area under the ROC curve" used to score every classifier and intrusion detector (Ch. 26) is ∫₀¹ TPR d(FPR), computed by the trapezoid rule over the curve's points. For points (0,0), (0.1,0.6), (0.3,0.85), (1,1):

AUC = 0.1·(0+0.6)/2 + 0.2·(0.6+0.85)/2 + 0.7·(0.85+1)/2
    = 0.030 + 0.145 + 0.6475 = 0.8225

from scipy.integrate import quad
import numpy as np
print(quad(lambda x: x**2, 0, 3))                       # (9.0, ...)  area under x²
print(quad(lambda x: np.exp(-x**2), -np.inf, np.inf))   # (1.7724..., ...) = √π
print(np.trapezoid([0, 0.6, 0.85, 1], [0, 0.1, 0.3, 1]))     # 0.8225  ROC AUC

Real-life engineering ties

• SWE: understanding "rate of change" dashboards (deploy velocity, error rate acceleration).
• AI: the derivative is literally what gradient descent follows downhill (Ch. 13, 21) — this chapter is the prerequisite for all of neural network training.
• Network: rate of change of queue depth or throughput signals congestion before it becomes an outage (used in AQM/CoDel algorithms, Ch. 25).
• Security: rate-of-change of failed login attempts (acceleration, not just count) is a stronger brute-force signal than a raw threshold.

More real-life examples:

• Cloud cost/capacity dashboards that show "spend rate" and "rate of change of spend" are literally reporting a metric and its derivative — a sudden jump in the derivative is a much earlier warning sign than watching total spend alone.
• Physics engines in games and robotics simulations integrate acceleration to get velocity and velocity to get position every frame — the same derivative chain (position → velocity → acceleration) run continuously at 60+ times a second.
• Newton's method, a numerical technique that uses a function's derivative to iteratively find where it crosses zero, is used inside many optimizers, root-finders, and implied-volatility calculations in finance.
• SRE teams track error-rate velocity (how fast the error rate itself is climbing) as an earlier, more sensitive incident-detection signal than a static error-rate threshold — a derivative-based alert, not a level-based one.

Common mistake

Confusing a rate with a total. "Requests per second is climbing" is a statement about the derivative; "we've served 1M requests today" is a statement about the integral (the accumulated total, Ch. 13). Dashboards that don't clearly label which one you're looking at cause real misdiagnosis during incidents.