Chapter 16 — Why "deep" matters

In one sentence

Depth lets a network build concepts out of simpler concepts, which is dramatically more efficient than trying to express everything at once.

The idea

Each layer composes the previous layer's outputs into something more abstract.

IMAGE NETWORK                          LANGUAGE MODEL (roughly)
-------------                          -----------------------
layer 1:  edges, blobs                 early:  which token is this? spelling
layer 2:  corners, textures            early-mid: grammar, short phrases
layer 3:  eyes, wheels, fur patches    middle: who does "she" refer to?
layer 4:  faces, cars                            facts, relationships
layer 5:  "cat", "sports car"          late:   task structure, what to say next

Nobody programmed this hierarchy. It emerges because it is the most efficient way to reduce the loss.

Why depth beats width: to recognise a face using one layer, you'd need a separate detector for every possible face-in-every-pose — combinatorially impossible. With layers, you build eyes once and reuse them across all faces. Composition is exponentially more efficient than enumeration.

The unlock: why 2012 and not 1992

The ideas were mostly there in the 1980s. Three things changed:

GPUs. Graphics cards do exactly the operation neural nets need (huge matrix multiplies) hundreds of times faster than CPUs. Training that took months took days.
Data. ImageNet (2009): 14 million labelled images. The internet made large datasets possible.
Tricks that made deep nets trainable. ReLU, better initialisation, dropout, batch normalisation, and later residual connections.

The moment: AlexNet, 2012. A deep convolutional network won the ImageNet competition by a huge margin over hand-engineered-feature methods. Within three years, essentially all computer vision was deep learning.

The "Bitter Lesson"

Rich Sutton's influential 2019 essay argues that across 70 years of AI, methods that leverage more computation consistently beat methods that leverage human knowledge — even though the human-knowledge approaches feel more satisfying and win in the short term.

Debated: many researchers push back, arguing that structure and priors still matter (and that "just scale it" has limits, especially data limits). Read the essay and the counter-arguments; both views inform how the field actually works.

Check yourself

Give an example of a feature hierarchy.
Why is depth more efficient than width?
What three things made deep learning practical around 2012?