Core6 min read

The vanishing past

By step 400, step 5 has had its say… and its gradient multiplied away to nothing. Why simple RNNs can't learn long-range dependencies.

Unfold the RNN loop and you get a deep chain — the removed chapter-2 problem back at sequence length. The loss usually arrives at the end (scoring the result), so gradient reaching step 5 in a 500-step sequence is a product of ~495 factors:

\frac{\partial h_L}{\partial h_t} \approx \prod_{\tau = t+1}^{L} W_h \cdot f'(\tau)

With $W_h$ 's spectral behavior and tanh derivatives ( $\le 1$ ) as the per-step factor, the product collapses — or, less commonly but worse, derails into growth. Either way: steps far from the loss receive nearly no learning signal.

The concrete failure

“The chef who trained in Paris made noodles and — at step 400, the promised successor was magnificent in the fridge”. Language needs the pronoun to label the chef; RNN gradients mostly can't route evidence from step 400 back to its own step, so the update tells more-step factors how to change by nudging late steps. Early evidence of named variables was all but imperceivable.

Gradient strength reaching each earlier time step (right = last step, left = far past).

per-step factor ∂h_t/∂h_t−10.93

usable memory horizon ≈ >60 steps

The gradient is below 1% of its strength after the shaded span (63 steps back). Nothing earlier than that is being influenced by what arrived — the mechanism behind untrainable long-range dependencies in plain RNNs.

|factor|^(distance back) on a log scale: the multiplicative anatomy when the same recurrence weights repeat at every unrolled step. Real curves wobble with W_h’s spectrum and gate states, but every honest long-RNN shows a usable horizon of tens of steps — this is why plain RNNs can’t learn long-range dependencies.

The two-standard-bends answer

Both classic rescues live in a trade your earlier lessons created:

Squeeze the input shorter. Attention-flavored models come later (your modern default — direct lookups); earlier canonical seq2seq process-to-brief uses this Band-Aid exactly.
Build memory better, not bigger. LSTM cells — next lesson.

Gradients existing and being informative are separate things; shortening sequences from SSM designs (today's alternative family) addresses the multiplicative decay rate directly by construction.

Vanishing ≠ the last layers

Escaping the recent past works fine (last 20–50 steps in tanh units) — that's within factor-distance of plain descent. What really fails is deep recurrence: correlations needing hundreds of steps survive only if the cell was trained to babysit them — and that instruction never reaches through distances this long.

Illustrative vs real

Simulation-like narrative sweep over a real gradient product in a synthetic chain. Real long-RNNs mix effects (correlations, skip connections, truncation heuristics, decoupled state) — yet are empirically trapped in exactly this shortened span; that operational finding replicates everywhere.

Where next: gates-and-lstms — an engineered cell that chooses, per step, whether to update the memory at all.

This lesson has exercises attached — locating the effective memory horizon — launching once the exercises layer ships.

The two-standard-bends answer #

The two-standard-bends answer