Chapter 17 — Networks that read sequences (RNNs) and why they struggled

Read one word, update your memory, repeat.

              h0        h1        h2        h3
              |         |         |         |
   [memory] --+-> [RNN] +-> [RNN] +-> [RNN] +-> ...
                   ^         ^         ^
                   |         |         |
                 "the"     "cat"     "sat"

At each step:  new_memory = f(old_memory, current_word)

Why plain RNNs failed, and the LSTM fix

Backpropagating through 50 steps means multiplying 50 Jacobians — the vanishing- gradient problem (Chapter 15) at its worst. Plain RNNs effectively forgot anything more than ~10 words back.

The LSTM (Long Short-Term Memory, 1997) added a cell state — a memory conveyor belt with additive updates and learned gates controlling what to erase, write, and read:

   +-------------------- cell state (the conveyor belt) -------------------->
        ^ erase?          ^ write?                    ^ read?
        |                 |                           |
    forget gate       input gate                  output gate

Because the cell state is updated by addition rather than repeated multiplication, gradients survive far longer. This is the same trick as a residual connection. LSTMs could remember hundreds of steps and dominated NLP from ~2014 to 2017.

The 2014 breakthrough that led to Transformers: attention

For translation, the design was sequence-to-sequence: one RNN reads the source sentence into a single vector, another RNN generates the translation from it.

The bottleneck: the entire source sentence has to be squeezed into one fixed vector. Long sentences got mangled.

Attention (Bahdanau et al., 2014): instead of one vector, let the decoder look back at all the encoder's hidden states at each output step, and learn which ones to focus on.

Translating "the cat sat" -> "le chat s'est assis"

When generating "chat", attention weights might be:
    the: 0.1     cat: 0.8     sat: 0.1
                  ^^^^ focus here

This worked so well that in 2017 researchers asked the obvious question: "if attention is doing the heavy lifting, do we need the RNN at all?"

The answer was no. That paper was called "Attention Is All You Need", and it is Part 6.

Why Transformers replaced RNNs

Speed was the decisive factor: a Transformer trains in hours what an LSTM needed weeks for, which meant researchers could try bigger models and more data — and that's what produced LLMs.

Not dead: RNN-like models are having a renaissance (Mamba, RWKV, state-space models) because they run inference in constant memory, unlike Transformers. Some current models are hybrids. But the LLM era was built on Transformers.