13. Calculus II — Gradients, Partial Derivatives, the Chain Rule

Level: Intermediate You'll use this for: the entire mathematical basis of how neural networks learn (backpropagation, Ch. 21).

Partial derivatives

When a function has multiple inputs, ∂f/∂x measures the rate of change with respect to x, holding every other variable fixed. For f(x, y) = x²y + y³:

∂f/∂x = 2xy          (treat y as a constant)
∂f/∂y = x² + 3y²      (treat x as a constant)

Getting there term by term. f(x,y) = x²y + y³ has two additive terms, so differentiate each separately (sum rule, Ch. 12) and add:

∂f/∂x  (treat y as a fixed number, like "5"):
  d/dx (x²y) = y · d/dx(x²) = y · 2x = 2xy    <- power rule on x, y is just a constant multiplier
  d/dx (y³)  = 0                               <- no x in this term at all, so its rate of change w.r.t. x is 0
  sum: ∂f/∂x = 2xy + 0 = 2xy

∂f/∂y  (treat x as a fixed number):
  d/dy (x²y) = x² · d/dy(y) = x² · 1 = x²      <- x² is just a constant multiplier
  d/dy (y³)  = 3y²                              <- power rule on y
  sum: ∂f/∂y = x² + 3y²

Every partial derivative is an ordinary single-variable derivative in disguise — you just freeze every other variable as if it were a plain number first.

The gradient

The gradient ∇f is the vector of all partial derivatives — it points in the direction of steepest ascent of f. Negate it, and you get the direction of steepest descent — the entire idea behind gradient descent (Ch. 21).

import sympy as sp
x, y = sp.symbols('x y')
f = x**2 * y + y**3
gradient = [sp.diff(f, x), sp.diff(f, y)]
print(gradient)   # [2*x*y, x**2 + 3*y**2]

The chain rule (the single most important rule in ML math)

If y = f(g(x)), then dy/dx = f'(g(x)) · g'(x). In words: "the rate of change of the outer function times the rate of change of the inner function." This is the entire mathematical mechanism behind backpropagation — a neural network is nothing but nested functions (layer₁(layer₂(layer₃(...)))), and computing how the final loss changes with respect to an early-layer weight requires applying the chain rule across every layer in between.

# y = (3x + 1)^2  =>  outer: u^2, inner: u = 3x+1
x = sp.symbols('x')
y = (3*x + 1)**2
print(sp.diff(y, x))          # 6*(3*x + 1)  -- sympy applies chain rule automatically

layer1: h = w1 * x
layer2: y = w2 * h
loss:   L = (y - target)^2

∂L/∂w1 = ∂L/∂y · ∂y/∂h · ∂h/∂w1

The gradient, symbol by symbol

∇f(x) = [ ∂f/∂x₁,  ∂f/∂x₂,  ...,  ∂f/∂xₙ ]

Example: for f(x,y) = x²y + y³ at the point (1, 2):

∇f = (2xy, x² + 3y²) = (2·1·2, 1 + 3·4) = (4, 13)

The output is about 3× more sensitive to y than to x here. In ML terms, y's weight would get a 3× bigger update.

Why the gradient points uphill (a two-line proof)

The slope in an arbitrary unit direction u is the directional derivative:

D_u f = ∇f · u = ‖∇f‖ · ‖u‖ · cos θ = ‖∇f‖ · cos θ

(This uses the geometric dot product from Ch. 10, with ‖u‖ = 1.) cos θ is largest (= 1) when θ = 0, that is, when u points exactly along ∇f. So the gradient is the direction of steepest ascent, and its length is that steepest slope. -∇f is steepest descent. Perpendicular to ∇f (cos θ = 0) the slope is zero, which is the direction of a contour line on a map.

Example: at (1, 2) with ∇f = (4, 13), moving in direction u = (0.6, 0.8) gives slope 4·0.6 + 13·0.8 = 12.8. Moving along the gradient itself gives ‖∇f‖ = √(16 + 169) ≈ 13.6, which is steeper, as the proof promised.

The Jacobian: derivatives of vector-valued functions

When a function takes a vector in and gives a vector out, as every neural network layer does, its derivative is a matrix:

       [ ∂y₁/∂x₁   ∂y₁/∂x₂   ...  ∂y₁/∂xₙ ]
J  =   [ ∂y₂/∂x₁   ∂y₂/∂x₂   ...  ∂y₂/∂xₙ ]        (m outputs × n inputs)
       [   ...                             ]
       [ ∂yₘ/∂x₁   ...            ∂yₘ/∂xₙ ]

Row i is the gradient of output i. The multivariable chain rule becomes matrix multiplication of Jacobians:

if  y = f(x)  and  L = g(y),  then   ∂L/∂x = Jᵀ · ∂L/∂y

So backpropagation is a sequence of matrix-vector products, running the forward pass's matrices in reverse, transposed. That's why a backward pass costs about 2× a forward pass and runs on the same GPU matrix units.

Backprop through a linear layer, by hand

For y = W·x (Ch. 10's layer, bias left out for clarity) with loss L, and writing δ = ∂L/∂y (the "error signal" arriving from above), the two gradients you need are:

∂L/∂W = δ · xᵀ          (an outer product: same shape as W)
∂L/∂x = Wᵀ · δ          (sent further back to the previous layer)

Same W and x as Ch. 10, target t = (0, 2), loss L = ½‖y - t‖²:

x = (1, 2)       W = [0.5  -1  ]       y = W·x = (-1.5, 2.5)
                     [2     0.25]

L = ½·((-1.5-0)² + (2.5-2)²) = ½·(2.25 + 0.25) = 1.25
δ = y - t = (-1.5, 0.5)                                    (derivative of ½(y-t)²)

∂L/∂W = δ·xᵀ = [-1.5] · [1  2] = [-1.5  -3.0]
               [ 0.5]            [ 0.5   1.0]

∂L/∂x = Wᵀ·δ = [0.5   2  ] · [-1.5] = [ 0.25 ]
               [-1    0.25]   [ 0.5]   [ 1.625]

The weight connecting input 2 (the larger input, 2) to output 1 (the larger error, -1.5) gets the biggest gradient, -3.0. Learning is assigning blame in proportion to activity × error. That's the mathematical form of Hebb's "neurons that fire together wire together". loss.backward() computes exactly these two products for every layer.

The most elegant gradient in deep learning: softmax + cross-entropy

Classifiers (and every LLM's next-token prediction) turn raw scores z ("logits") into probabilities with softmax, then score them with cross-entropy against the true class c:

pᵢ = e^(zᵢ) / Σⱼ e^(zⱼ)                L = -ln(p_c)

Substitute and simplify: L = -z_c + ln Σⱼ e^(zⱼ). Differentiate with respect to zᵢ:

∂L/∂zᵢ = -[i = c] + e^(zᵢ)/Σⱼ e^(zⱼ)  =  pᵢ - yᵢ

where yᵢ is 1 for the true class and 0 otherwise (one-hot). After all those exponentials and logs, the gradient is just "predicted minus actual".

Example: logits z = (2.0, 1.0, 0.1), true class 0:

p = softmax(z) = (0.659, 0.242, 0.099)
L = -ln(0.659) = 0.417
∂L/∂z = p - y = (0.659-1, 0.242, 0.099) = (-0.341, 0.242, 0.099)

Push the correct logit up, push the wrong ones down, each in proportion to how wrong it was. Every token of every LLM's pre-training uses this gradient, trillions of times over.

The Hessian: curvature in many dimensions

The matrix of all second partial derivatives:

H[i][j] = ∂²f / ∂xᵢ ∂xⱼ

It's symmetric (the order of differentiation doesn't matter for smooth f, which is Schwarz's theorem), so by the spectral theorem (Ch. 11) its eigenvalues are real. They give the second-derivative test in n dimensions. At a point where ∇f = 0:

Examples:

f(x,y) = x² + xy + y²   ->   H = [2 1]   eigenvalues 1, 3  (Ch. 11!)  ->  minimum at (0,0)
                                 [1 2]
f(x,y) = x² - y²        ->   H = [2  0]  eigenvalues 2, -2            ->  saddle at (0,0)
                                 [0 -2]

Why this matters for deep learning: in a model with a billion parameters, a critical point is a minimum only if all billion eigenvalues are positive. If signs were coin flips, that would be absurdly unlikely, so most points where the gradient vanishes are saddles, not bad local minima (Dauphin et al., 2014). Plain gradient descent slows to a crawl near saddles, and momentum (Ch. 21) helps carry it through. The Hessian's largest eigenvalue also sets the maximum stable learning rate, η < 2/λ_max, which Ch. 21 derives.

Putting it together, the multivariable Taylor expansion (Ch. 12, in n dimensions) is the model behind every optimizer:

f(x + Δ) ≈ f(x)  +  ∇f(x)ᵀ·Δ  +  ½ · Δᵀ·H·Δ
           value     slope         curvature

Gradient descent uses the first two terms. Newton's method and its approximations (L-BFGS, K-FAC, Shampoo) use all three.

Networking in the wild: splitting traffic by marginal delay

Two backends: a fast one (μ₁ = 100 req/s) and a slow one (μ₂ = 50 req/s). Total traffic λ = 90 req/s. Send x to the fast one and 90 - x to the slow one. By Little's Law and the M/M/1 formula (Ch. 25), the average number of requests in the system is:

N(x) = x/(μ₁ - x) + (λ - x)/(μ₂ - (λ - x))         average delay = N / λ

Set the derivative to zero. d/dx [x/(μ - x)] = μ/(μ - x)², so the optimum balances marginal delays, not loads:

μ₁/(μ₁ - x)² = μ₂/(μ₂ - λ + x)²    ->   x* ≈ 64.9 req/s to the fast server

Proportional-to-capacity looks "obviously fair", but the derivative says to overload the fast server slightly, because its queue grows more slowly. This is Gallager's 1977 minimum-delay routing: at the optimum, every path in use has equal marginal delay ∂D/∂xᵢ. It's the same "equalize the derivatives" condition as the fleet-sizing example in Ch. 12, and the same idea behind least-loaded and "power of two choices" load balancing in Envoy and NGINX.

from scipy.optimize import minimize_scalar
mu1, mu2, lam = 100, 50, 90
N = lambda x: x/(mu1-x) + (lam-x)/(mu2-(lam-x))
best = minimize_scalar(N, bounds=(40.01, 89.99), method="bounded")
print(best.x, 1000 * N(best.x) / lam)    # ~64.85 req/s, ~31.7 ms

Real-life engineering ties

• AI: this chapter is the prerequisite for understanding backpropagation and automatic differentiation (autograd/.backward() in PyTorch) rather than treating it as a magic black box.
• SWE: understanding sensitivity analysis — "how much does output change if this input changes slightly" — for tuning any system with multiple interacting parameters.
• Network/Security: less direct, but the same "how sensitive is the output to a small input change" question underlies stability analysis of control loops (Ch. 29) like TCP congestion control and rate limiters.

More real-life examples:

• Options pricing in finance defines "the Greeks" (Delta, Gamma, Vega) as partial derivatives of an option's price with respect to the underlying stock price, volatility, and time — the same partial-derivative machinery as this chapter, applied to markets instead of models.
• PyTorch's and TensorFlow's autograd engines build a computation graph of every tensor operation and apply the chain rule automatically, backward through the graph, to compute every gradient needed for training — this chapter is literally what .backward() does under the hood (Ch. 21 walks through it by hand).
• Distributed-systems performance engineers ask chain-rule-flavored sensitivity questions constantly: "if the auth service gets 10ms slower, how much slower is the end-to-end checkout request" — the answer depends on how many chained calls sit between the two, exactly like nested functions in the chain rule.
• Chemical and biological process control systems use partial derivatives of a target variable (e.g., reactor temperature) with respect to multiple inputs (flow rate, pressure) to decide which knob to turn — the same gradient idea used to train a neural net (Ch. 21), applied to a physical process.

Common mistake

Forgetting to multiply by the inner derivative — computing only f'(g(x)) and stopping, instead of f'(g(x)) · g'(x). This single dropped term is the most common manual backpropagation error.