Chapter 10 — How a model learns: loss and gradient descent

In one sentence

Define a number that measures wrongness (loss), then repeatedly step every knob slightly downhill on that number.

The idea, in plain language

You're on a foggy hillside and want to reach the valley. You can't see far, but you can feel which way the ground slopes under your feet. So: feel the slope, take a small step downhill, repeat.

   loss
    ^
    |  \                              /
    |   \                            /
    |    \      you start here      /
    |     \        o               /
    |      \       |              /
    |       \      v  step        /
    |        \     o             /
    |         \    |            /
    |          \   v           /
    |           \  o          /
    |            \_|_ o _ o _/     <- the minimum: lowest loss
    |
    +--------------------------------------> value of one knob

The hillside = the loss as a function of your knobs.
The slope = the gradient.
Step size = the learning rate.
Doing this = gradient descent.

How it actually works

The loss for classification: cross-entropy.

You want to punish confident wrong answers heavily. Cross-entropy does this:

loss for one example = -log( probability the model gave to the CORRECT answer )

Model's probability for the truth	Loss
0.99 (confident and right)	0.01 — tiny
0.50 (unsure)	0.69
0.10 (wrong)	2.30
0.01 (confidently wrong)	4.61 — huge
0.001	6.91 — enormous

That asymmetry is the point: being confidently wrong is much worse than being unsure. Remember this table — the identical loss trains every LLM (Chapter 33).

The gradient. For each knob, the gradient answers: "if I increase this knob a tiny bit, does the loss go up or down, and how fast?" You then move each knob the opposite way.

for each knob w:
    w = w - learning_rate * (slope of loss with respect to w)

Computing those slopes for millions of knobs efficiently is backpropagation (Chapter 15). Libraries do it for you.

Variants you'll hear:

Name	What it means
Batch gradient descent	compute the gradient using all data, then step. Accurate, slow.
Stochastic GD (SGD)	use one example per step. Fast, noisy.
Mini-batch SGD	use 32–4096 examples per step. This is what everyone uses.
Epoch	one full pass over the training data
Learning rate	step size. The most important setting you will ever tune.
Adam / AdamW	smarter step rules that adapt per-knob. AdamW trains every LLM.

Worked example: watch the loss table

Run this and read the numbers:

import numpy as np

def sigmoid(z): return 1 / (1 + np.exp(-z))

# 4 emails, 2 features: [count_free, count_meeting]
X = np.array([[2,0],[3,0],[0,2],[0,3]], dtype=float)
y = np.array([1, 1, 0, 0])                      # spam, spam, ham, ham

w = np.zeros(2)                                  # start with no knowledge
b = 0.0
lr = 0.5

for step in range(101):
    z = X @ w + b
    p = sigmoid(z)
    loss = -np.mean(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))

    # gradients (derived from the cross-entropy loss)
    dz = p - y
    dw = X.T @ dz / len(y)
    db = dz.mean()

    w -= lr * dw
    b -= lr * db

    if step % 20 == 0:
        print(f"step {step:3d}  loss {loss:.4f}   w_free={w[0]:+.2f}  "
              f"w_meeting={w[1]:+.2f}  b={b:+.2f}")

print("\nfinal predictions:", sigmoid(X @ w + b).round(3))

Watch w_free climb positive and w_meeting fall negative — the model is learning "free = spammy, meeting = not". You can now see learning happen.

Practice (20 min)

Change lr to 5.0. Then 0.001. Describe both failures in notes.md.
Add a contradictory example: X = [[2,0]], y = [0] (an email with "free" that is not spam). What happens to the final loss? (It can no longer reach zero — the data is not separable. This is normal and important.)
Print p (the probabilities) every 20 steps instead of the weights. Watch confidence grow.

Common confusions

"Does gradient descent find the best answer?" For logistic regression, yes (the loss has one valley). For neural networks, no — but it reliably finds a good enough valley, which is one of the happy surprises of the field.
"Why not just try all knob values?" With 8 billion knobs and 10 values each, that's 10^8000000000 combinations. Gradient descent is the only way.
Loss vs accuracy: loss is what you optimise (smooth, differentiable); accuracy is what you report (jumpy, not differentiable). They usually move together, but not always.

Check yourself

What is the loss and what is the gradient?
Why is confidently-wrong punished so much by cross-entropy?
What happens if the learning rate is too high? Too low?