Chapter 15 — How neural networks learn: backpropagation, gently

In one sentence

Backpropagation is the chain rule from calculus, applied backwards through the network, so every knob learns how much it contributed to the error.

The problem

Gradient descent (Chapter 10) needs to know, for each of possibly billions of knobs, "if I nudge this one, does the loss go up or down?" Computing that separately for each knob would take forever. Backprop computes all of them in one backward sweep, costing about twice a forward pass.

The idea, in plain language: assigning blame

A restaurant serves a bad dish. Who's to blame?

FORWARD (making the dish):
  ingredients --> prep cook --> sauce chef --> plating --> customer: "too salty!"

BACKWARD (assigning blame):
  customer complaint "too salty"
       |
       v  how much did plating contribute to saltiness?  -> a little
  plating
       |
       v  how much did the sauce contribute?             -> a lot
  sauce chef
       |
       v  how much did the prep contribute?              -> some
  prep cook

Each station learns HOW MUCH TO CHANGE, based on
  (how much the next station blamed them)  x  (how sensitive they are)

That multiplication — blame from downstream × local sensitivity — is exactly the chain rule.

How it actually works

The chain rule says: if a affects b, and b affects c, then

   how much a affects c  =  (how much a affects b)  x  (how much b affects c)

In a network, the loss depends on the output, which depends on layer 3, which depends on layer 2, and so on. So you compute the sensitivity at the output and multiply your way backwards.

FORWARD PASS  (compute the answer)
  x --[W1]--> z1 --[relu]--> h1 --[W2]--> z2 --[softmax]--> p --> LOSS
  ============================================================>

BACKWARD PASS (compute all gradients)
  <============================================================
  dW1 <-- dz1 <-- dh1 <-- dW2 <-- dz2 <-- dp <-- dLOSS

  At each step you need only:
     (a) the gradient flowing in from the right
     (b) the values you saved during the forward pass

That's why frameworks store activations during the forward pass — they're needed for the backward pass. (It's also why training uses much more memory than inference.)

The one formula worth remembering — for a linear layer out = W @ in:

gradient for the weights   =  (gradient flowing in)  x  (the layer's input)
gradient to pass leftward  =  W^T  x  (gradient flowing in)

"Gradient for a weight = upstream blame × what that weight was multiplying." That's intuitive: a weight that multiplied a big input has a big effect, so it gets a big gradient.

Worked example: two knobs, all the arithmetic

Model: y = w2 * (w1 * x). Input x = 2, target t = 10, start w1 = 1, w2 = 1. Loss = (y - t)^2.

FORWARD:
  a  = w1 * x = 1 * 2  = 2
  y  = w2 * a = 1 * 2  = 2
  loss = (2 - 10)^2 = 64

BACKWARD:
  dloss/dy  = 2*(y - t) = 2*(2-10) = -16

  dloss/dw2 = dloss/dy * dy/dw2 = -16 * a  = -16 * 2 = -32
  dloss/da  = dloss/dy * dy/da  = -16 * w2 = -16 * 1 = -16
  dloss/dw1 = dloss/da * da/dw1 = -16 * x  = -16 * 2 = -32

UPDATE (learning rate 0.01):
  w2 = 1 - 0.01*(-32) = 1.32
  w1 = 1 - 0.01*(-32) = 1.32

CHECK (forward again):
  a = 1.32*2 = 2.64 ;  y = 1.32*2.64 = 3.48 ;  loss = (3.48-10)^2 = 42.4
  Loss went from 64 to 42.4.  It learned.

Do this once with a pen. It removes all the mystery from backpropagation.

Practice (25 min)

A. Verify gradients numerically. The safest way to know your maths is right:

import numpy as np

def f(w1, w2, x=2.0, t=10.0):
    return (w2 * (w1 * x) - t) ** 2

w1, w2, x, t = 1.0, 1.0, 2.0, 10.0

# analytic gradients (what we derived above)
a = w1 * x
y = w2 * a
g_w2 = 2*(y - t) * a
g_w1 = 2*(y - t) * w2 * x

# numerical gradients: (f(w+eps) - f(w-eps)) / (2*eps)
eps = 1e-6
n_w1 = (f(w1+eps, w2) - f(w1-eps, w2)) / (2*eps)
n_w2 = (f(w1, w2+eps) - f(w1, w2-eps)) / (2*eps)

print(f"w1: analytic {g_w1:.4f}   numeric {n_w1:.4f}")
print(f"w2: analytic {g_w2:.4f}   numeric {n_w2:.4f}")

They should match to ~4 decimal places. This technique will save you hours whenever you hand-write a backward pass.

B. Let PyTorch do it and see that it agrees:

import torch

w1 = torch.tensor(1.0, requires_grad=True)
w2 = torch.tensor(1.0, requires_grad=True)
x, t = torch.tensor(2.0), torch.tensor(10.0)

loss = (w2 * (w1 * x) - t) ** 2
loss.backward()                      # <-- backprop, one line

print("loss:", loss.item())
print("w1.grad:", w1.grad.item())    # should be -32
print("w2.grad:", w2.grad.item())    # should be -32

This is what every deep-learning framework does for you. You never write backprop by hand in real work — but you must know what it's doing to debug anything.

Two problems you will hear about

Vanishing gradients. Multiply many numbers smaller than 1 and you get something microscopic. In a deep network, the gradient reaching layer 1 can be ~0, so early layers never learn. Sigmoid activations made this terrible.

Exploding gradients. The opposite: numbers > 1 multiplied many times explode to infinity, and the loss becomes NaN.

The fixes (all used in LLMs):

Fix	What it does
ReLU/GELU activations	don't squash the signal for positive inputs
Careful initialisation	start weights at a scale that preserves signal size
Normalisation layers (LayerNorm/RMSNorm)	rescale activations back to a sane range at each layer
Residual connections `y = x + F(x)`	give the gradient a direct highway past each layer — this is why 100-layer networks are trainable at all
Gradient clipping	cap the gradient size so one bad batch can't blow up the weights

Residual connections deserve emphasis: they are in every Transformer block you'll meet in Part 6. Without them, deep Transformers would not train.

Common confusions

"Do I need to understand the calculus?" To use deep learning: no. To debug it: understanding the picture (blame flowing backwards, gradients vanishing/exploding) is essential. The formulas, no.
"Backprop is the learning algorithm" — not quite. Backprop computes gradients; gradient descent (or Adam) uses them to update. Two separate things.

Check yourself

Explain backpropagation using the restaurant analogy.
Why do frameworks store activations during the forward pass?
What is a vanishing gradient, and name two fixes.
What does a residual connection do for the gradient?