Wiki
Core7 min read

Gates and LSTMs

The LSTM decides, at every step, what to remember and what to overwrite — three learned gates that kept memory stable for hundreds of steps.

The previous lesson located the failure precisely: uncontrolled overwriting. The plain RNN cell rewrites its whole state at every step, so each step's rewriting compounds. The fix ( Hochreiter & Schmidhuber, 1997) is surgical: add three learned gates between the memory and each incoming step, so the cell can choose not to touch the memory at all.

ft=σ(Wfxt+Ufht−1)forget gateit=σ(Wixt+Uiht−1)input gateot=σ(Woxt+Uoht−1)output gatect=ft⊙ct−1+it⊙c~tht=ot⊙tanh⁡(ct)\begin{aligned} f_t &= \sigma(W_f x_t + U_f h_{t-1}) && \text{forget gate} \\ i_t &= \sigma(W_i x_t + U_i h_{t-1}) && \text{input gate} \\ o_t &= \sigma(W_o x_t + U_o h_{t-1}) && \text{output gate} \\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\ h_t &= o_t \odot \tanh(c_t) \end{aligned}

The load-bearing line is ctc_t: forget-gate-controlled carry (ft⊙ct−1f_t \odot c_{t-1}) plus gated incoming candidate c~t\tilde{c}_t. When ft≈1f_t \approx 1 and it≈0i_t \approx 0, state flows through effectively unchanged — the multiplicative chain the main gradient travels through is a near-identity:

∂ct∂ct−1=ft(≈1 when not updating) \frac{\partial c_t}{\partial c_{t-1}} = f_t \quad (\approx 1 \text{ when not updating})

Memory that doesn't have to be touched

A plain RNN is a whiteboard someone wipes and rewrites once per word. An LSTM is a whiteboard with a latch on each word's write: unless the gate says this word matters, it can't smear what's stored. The gradient highway is preserved by not reusing the weight matrix at every step against a fickle function, but by an additive state ride gated by a learned, near-continuous switch. (That additive state ride is the same idea as residual connections, years earlier, in sequence form.)

One LSTM cell reading “the chef, who trained in Paris, was …” — a long-range subject–verb dependency.

thechefwhotrainedinparis,was
forget f
0.97
input i
0.15
output o
0.60
cell c
0.88

clause content — memory rides unchanged (f≈1, i≈0)

Step 1: the *chef* writes itself into memory, then gates clamp shut through the relative clause — f≈1, i≈0 is the whole trick. At “was”, forget opens hard, and the output gate reads the cached subject: no multiplicative wear across the seven steps of no-update.

Real gate/state arithmetic per the LSTM equations, driven on a one-dimensional agreement track. Illustrative: gate values here are hand-set to tell the true story an *aligned-to-language* trained LSTM produces; their learning of which words deserve writes/reads is what training the other two gates handles.

What it actually fixed — and what it couldn't

LSTMs ran language modeling for a decade (translated streams, speech-modeling pre-transformer, the attention handoff coming next were its products). The constant catch:

  • Still sequential. Can't process steps in parallel — training spends step-time in real wall-clock, and big data makes serial runs expensive regardless of gate quality.
  • Still bottleneck-bearing. The whole past of the sentence is one vector, no matter how well-gated; this is why long segment aggregation eventually displaced the whole family.

GRU — a two-gate, streamlined cousin — trades mid-strength memory for fewer parameters and trains faster; in usage it's the "small batch" instance of this family and still appears mid-scale where cost matters in today's production flows.

Illustrative vs real

Real gate arithmetic from the equations above on a tiny two-unit cell, driven through an illustrative sentence; gates shown as bar states. Missing at toy scale: the trained calibrations that make real LSTMgate schedules meaningful across a full corpus — show what the machinery reads as.

Check yourself

Eduspheria wiki · Deep Learning, Sequences before attention

0 / 5 answered

  1. 1In the LSTM cell, what does the forget gate f_t control?
    Multiple choice
  2. 2When the forget gate is near 1 and the input gate is near 0, the LSTM cell state passes through almost unchanged.
    True / false
  3. 3The LSTM's additive cell-state update mirrors which earlier architecture idea from the vision chapter?
    Short answer
  4. 4With previous state 4, forget gate 0.5, input gate 0.5 and candidate 2, what is the new cell state c_t = f times c_prev + i times candidate?
    Numeric answer
  5. 5Which two-gate streamlined cousin of the LSTM is cheaper and trains faster?
    Short answer

From the assignment paper

Exam-modeled · Assignment/Quiz

0 / 5 answered

  1. 1In an HMM tagging model the transition probability from noun to verb is 0.2 and from noun to noun is 0.7, and the emission probability of the word flies as a verb is 0.6. Which tag sequence is most likely for the pair time flies?
    Multiple choice
  2. 2In an HMM, what does a transition probability represent?
    Multiple choice
  3. 3Naive Bayes gives P(Play) = 0.6, P(Rainy given Play) = 0.2 and P(Rainy) = 0.4. What is P(Play given Rainy)?
    Numeric answer
  4. 4In an HMM, a zero emission probability for a word and tag pair makes that sequence impossible under the model.
    True / false
  5. 5Which probability does a classifier compute from the likelihood and the class prior, according to the assignment quiz?
    Short answer

Where next: the-attention-bridge — the honest arrival: why the whole approach dissolves into a lookup, and why every modern network uses the dissolver.

This lesson has exercises attached — predicting state survival under gate configurations — launching once the exercises layer ships.