Part 4 — Era 3: Neural networks (2010s)

Part 3 pushed hand-designed features — word counts, TF-IDF, n-gram tables — about as far as they go, and Chapter 13 showed you how to tell when a model is actually working. This part removes the ceiling: networks that learn their own features straight from data, instead of you designing them by hand. That single shift is what makes every later part of this guide possible.

Chapter 14 — What a neural network actually is

In one sentence

A neural network is logistic regression stacked in layers, so that the model learns its own features instead of you designing them.

The problem with logistic regression

Your spam filter draws a straight line through the data. Some problems can't be split by a straight line. The classic example is XOR:

   Can a straight line separate O from X?

    1 |  O           X                YES for AND/OR:
      |                                  1 |  O      X
      |                                    |    \
    0 |  X           O                   0 |  O  \   X
      +---------------                     +-------\-------
        0           1                        0      \    1
                                                 (a line works)
      XOR: no single straight line
      can put both O's on one side.

In 1969, Minsky and Papert proved a single-layer network (a "perceptron") could not learn XOR. Funding collapsed. The fix — adding a hidden layer — was known in principle, but nobody had a practical way to train it until backpropagation was popularised in 1986, and it didn't become practical at scale until GPUs arrived around 2010.

The idea, in plain language

Add a middle layer. The first layer invents new features; the second layer uses them.

INPUT LAYER          HIDDEN LAYER           OUTPUT LAYER
(your features)      (LEARNED features)     (the answer)

  x1  ------\        +-------+
             \------>|  h1   |------\
  x2  --------\----->|       |       \      +--------+
               \     +-------+        ----->| output |---> probability
  x3  ----------\--->+-------+       /      +--------+
                 \-->|  h2   |------/
  x4  -------------->|       |
                     +-------+

Each arrow has a WEIGHT (a knob).
Each box computes:  output = activation( sum of (weight x input) + bias )

Each hidden unit is its own little logistic regression over the inputs.
The network decides for itself what those hidden features should represent. Nobody tells it "detect an edge" or "detect sarcasm" — it discovers whatever is useful for reducing the loss.

That is the whole revolution of deep learning: features are learned, not designed.

The activation function (and why it's essential)

Between layers you apply a simple non-linear function. Without it, stacking layers is pointless:

Two linear layers with NO activation:
   y = W2 (W1 x)  =  (W2 W1) x  =  W x       <-- still just ONE linear layer!

With an activation in between:
   y = W2 ( relu(W1 x) )                     <-- genuinely more expressive

The common ones:

Name	Formula	Notes
Sigmoid	`1/(1+e^-x)`	Squashes to 0–1. Old. Causes vanishing gradients. Now mostly for output probabilities.
Tanh	squashed to −1..1	Better than sigmoid, still saturates.
ReLU	`max(0, x)`	"If negative, output 0; else pass through." Absurdly simple, hugely effective. Made deep nets trainable.
GELU	a smooth ReLU	The default in Transformers/BERT/GPT.
SwiGLU	a gated variant	Used inside modern LLMs (Llama, etc.).

ReLU in one picture:

       |      /
       |     /
       |    /
       |   /
   ----+--/-------
       | /
  0 0 0|/           output is 0 for all negative input,
       /            then rises linearly

Worked example: XOR by hand

Here is a network that solves XOR exactly. I'll just give you the weights; training would discover them on its own.

Inputs (x1, x2). Two hidden units h1, h2 using ReLU. One output o.

  h1 = relu( x1 + x2      )     -> "how many inputs are on"
  h2 = relu( x1 + x2 - 1  )     -> "how much MORE than one input is on"
  o  = h1 - 2 * h2

Now check all four cases:

  x1=0, x2=0:  h1 = relu(0) = 0    h2 = relu(-1) = 0    o = 0 - 2*0 = 0   want 0  OK
  x1=1, x2=0:  h1 = relu(1) = 1    h2 = relu( 0) = 0    o = 1 - 2*0 = 1   want 1  OK
  x1=0, x2=1:  h1 = relu(1) = 1    h2 = relu( 0) = 0    o = 1 - 2*0 = 1   want 1  OK
  x1=1, x2=1:  h1 = relu(2) = 2    h2 = relu( 1) = 1    o = 2 - 2*1 = 0   want 0  OK

All four correct.

Notice what the hidden layer invented: h1 counts how many inputs are on, and h2 fires only when both are on. Neither is anything you asked for — they are intermediate concepts the network needed. The output layer then says "one on, but subtract off the both-on case", which no single straight line could express.

Train it (Practice below) and it will find weights like these by itself. That is the whole point of a hidden layer.

Practice (30 min) — watch a network learn XOR

First, the visual version (5 min, no code): open playground.tensorflow.org in your browser.

Choose the XOR dataset (the checkerboard, second from left).
Remove all hidden layers -> press play. It cannot solve it.
Add one hidden layer with 4 neurons -> press play. It solves it in seconds.
Watch the little pictures inside each neuron: those are the learned features.

Then, the code version:

import numpy as np

X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y = np.array([[0],[1],[1],[0]], dtype=float)      # XOR

rng = np.random.default_rng(0)
W1 = rng.normal(0, 1, (2, 4));  b1 = np.zeros(4)     # 2 inputs -> 4 hidden
W2 = rng.normal(0, 1, (4, 1));  b2 = np.zeros(1)     # 4 hidden -> 1 output

def sigmoid(z): return 1/(1+np.exp(-z))
lr = 0.5

for step in range(20001):
    # ---- forward ----
    z1 = X @ W1 + b1
    h  = np.maximum(0, z1)                 # ReLU
    z2 = h @ W2 + b2
    p  = sigmoid(z2)
    loss = -np.mean(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))

    # ---- backward (chain rule; Chapter 15 explains it) ----
    dz2 = (p - y) / len(X)
    dW2 = h.T @ dz2;        db2 = dz2.sum(0)
    dh  = dz2 @ W2.T
    dz1 = dh * (z1 > 0)                    # ReLU derivative: 1 if input was >0
    dW1 = X.T @ dz1;        db1 = dz1.sum(0)

    W2 -= lr*dW2; b2 -= lr*db2; W1 -= lr*dW1; b1 -= lr*db1

    if step % 4000 == 0:
        print(f"step {step:5d}  loss {loss:.4f}  preds {p.ravel().round(3)}")

print("\nfinal predictions:", p.ravel().round(3), " (want [0, 1, 1, 0])")
print("\nlearned hidden features for each input:")
print(np.maximum(0, X @ W1 + b1).round(2))

Then experiment:

Set the hidden layer to 1 neuron ((2,1) and (1,1)). It fails. That's Minsky & Papert's proof, live.
Remove the ReLU (replace np.maximum(0, z1) with z1). It fails — because two stacked linear layers are still linear.
Print the hidden activations for each of the 4 inputs. What concept did each neuron learn?

Common confusions

"Is it like a brain?" Very loosely, and the analogy misleads more than it helps. It's a big differentiable function. Real neurons are far more complex.
"How many layers/neurons do I need?" There's no formula. Start small, increase until validation loss stops improving.
"Universal approximation" — a one-hidden-layer network can approximate any continuous function given enough neurons. This is a mathematical existence result; it doesn't tell you how many neurons, or that training will find them. In practice, depth is far more parameter-efficient than width, which is why we build deep networks.

Check yourself

Why can't a single-layer model solve XOR?
What breaks if you remove the activation function?
What does "the network learns its own features" mean?