The Wiki
Core6 min read

Recurring state

Same weights, looping over time: the RNN idea that let a fixed-size network read arbitrary-length sequences.

Sequences broke the input contract that made everything in chapter 1 tractable: a fixed set of inputs per forward pass. Sentences, audio, sensor logs — variable length, and order-sentitive. The re-interpretation that worked for a decade is almost embarrassingly small: run the same cell over the sequence one step at a time, and let it carry memory between steps.

ht=tanh⁡(Wxxt+Whht−1+b) h_t = \tanh(W_x x_t + W_h h_{t-1} + b)

One cell — the same WW at every time step (weight-sharing with time as its stamp axis, exactly like convolutions share across space). hth_t is the running state: a compressed summary of everything so far.

A reader with a working memory

Read a sentence one word at a time while carrying a hurry-up paraphrase in your head; after each word, refresh the paraphrase from the word-on-table plus your old paraphrase. That paraphrase is hth_t. The network's whole understanding of the past must physically fit in that one vector — keep this constraint in mind; it is the load-bearing wall of the next lesson's collapse.

One recurrent cell (2 hidden units), one real sentence, same weights every step.

thex1
chefx2
madex3
noodlesx4
withx5
couragex6
state (2-D toy memory)

cell state: (0.00, 0.00)

Every word runs through the same cell — same W — and its output *is* the next step’s memory input. Watch the state path bend as each word lands: the entire past of the sentence is whatever dot-chain ended up here.

Real recurrence: h_t = tanh(0.9·h + 0.6·x), two dampening channels, one shared cell — a miniature of the real loop. A real RNN runs 512+ dims with richer gates and a decoder read at the end; the walk-in-arrows shape is the true mechanism.

What the shared cell buys

  • Any length — the loop runs over 10 tokens or 10,000 with an unchanged parameter count.
  • Order sensitivity — swapping two input steps changes both feeds to the cell; nothing is permutation-invariant here.
  • Where the vision analogy differs — convs share across space but their outputs don't feed each other; the RNN's recurrence makes step tt's output an input of step t+1t+1, and that chaining is what makes training on sequences a special problem.

Unrolling

The standard mental model: unroll the loop — draw the same cell repeated, one box per time step, each feeding the next. Unrolling makes it a deep network of length LL with shared weights, and everything about gradient depths from chapter 2 comes roaring back — with the twist that the same weights appear in every factor of the product.

Illustrative vs real

The artifact runs a real 2-unit recurrent cell, artificially set up: a tangible readout of cell state at symbolic steps. Real RNNs alternate hidden dims (128–2048) and gate variants run under a training loop; the shape of state computation — update at every step, exact same cell — is faithful.

Where next: the-vanishing-past — what happens when the memory has to travel hundreds of steps without supervision arriving yet.

This lesson has exercises attached — tracing state updates step by step — launching once the exercises layer ships.