Core7 min read

Gates and LSTMs

The LSTM decides, at every step, what to remember and what to overwrite — three learned gates that kept memory stable for hundreds of steps.

The previous lesson located the failure precisely: uncontrolled overwriting. The plain RNN cell rewrites its whole state at every step, so each step's rewriting compounds. The fix ( Hochreiter & Schmidhuber, 1997) is surgical: add three learned gates between the memory and each incoming step, so the cell can choose not to touch the memory at all.

\begin{aligned} f_t &= \sigma(W_f x_t + U_f h_{t-1}) && \text{forget gate} \\ i_t &= \sigma(W_i x_t + U_i h_{t-1}) && \text{input gate} \\ o_t &= \sigma(W_o x_t + U_o h_{t-1}) && \text{output gate} \\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\ h_t &= o_t \odot \tanh(c_t) \end{aligned}

The load-bearing line is $c_t$ : forget-gate-controlled carry ( $f_t \odot c_{t-1}$ ) plus gated incoming candidate $\tilde{c}_t$ . When $f_t \approx 1$ and $i_t \approx 0$ , state flows through effectively unchanged — the multiplicative chain the main gradient travels through is a near-identity:

\frac{\partial c_t}{\partial c_{t-1}} = f_t \quad (\approx 1 \text{ when not updating})

Memory that doesn't have to be touched

A plain RNN is a whiteboard someone wipes and rewrites once per word. An LSTM is a whiteboard with a latch on each word's write: unless the gate says this word matters, it can't smear what's stored. The gradient highway is preserved by not reusing the weight matrix at every step against a fickle function, but by an additive state ride gated by a learned, near-continuous switch. (That additive state ride is the same idea as residual connections, years earlier, in sequence form.)

One LSTM cell reading “the chef, who trained in Paris, was …” — a long-range subject–verb dependency.

thechefwhotrainedinparis,was

forget f

0.97

input i

0.15

output o

0.60

cell c

0.88

clause content — memory rides unchanged (f≈1, i≈0)

Step 1: the *chef* writes itself into memory, then gates clamp shut through the relative clause — f≈1, i≈0 is the whole trick. At “was”, forget opens hard, and the output gate reads the cached subject: no multiplicative wear across the seven steps of no-update.

Real gate/state arithmetic per the LSTM equations, driven on a one-dimensional agreement track. Illustrative: gate values here are hand-set to tell the true story an *aligned-to-language* trained LSTM produces; their learning of which words deserve writes/reads is what training the other two gates handles.

What it actually fixed — and what it couldn't

LSTMs ran language modeling for a decade (translated streams, speech-modeling pre-transformer, the attention handoff coming next were its products). The constant catch:

Still sequential. Can't process steps in parallel — training spends step-time in real wall-clock, and big data makes serial runs expensive regardless of gate quality.
Still bottleneck-bearing. The whole past of the sentence is one vector, no matter how well-gated; this is why long segment aggregation eventually displaced the whole family.

GRU — a two-gate, streamlined cousin — trades mid-strength memory for fewer parameters and trains faster; in usage it's the "small batch" instance of this family and still appears mid-scale where cost matters in today's production flows.

Illustrative vs real

Real gate arithmetic from the equations above on a tiny two-unit cell, driven through an illustrative sentence; gates shown as bar states. Missing at toy scale: the trained calibrations that make real LSTMgate schedules meaningful across a full corpus — show what the machinery reads as.

Where next: the-attention-bridge — the honest arrival: why the whole approach dissolves into a lookup, and why every modern network uses the dissolver.

This lesson has exercises attached — predicting state survival under gate configurations — launching once the exercises layer ships.

What it actually fixed — and what it couldn't #

What it actually fixed — and what it couldn't