Gradient flow
The gradient tells every layer which way down is. In a deep net that signal is a long product — small factors kill it, large ones blow it up. This is *the* obstacle of depth.
Backpropagation from the ML book and the LLM book's training chapter is the chain rule run backwards:
Look at what that is for an -layer network: a product of factors. Products are unforgiving. Twenty factors of 0.5 is about a millionth. Twenty factors of 2 is about a million. The gradient reaching layer 1 is either dust or dynamite, and both are training failures — dust because early layers never move, explosion because every step flings parameters off the cliff.
The relay race picture
Imagine a 100-person relay where the message is a number that gets multiplied at every leg by that runner's attenuation. If runners pass 0.9 each, the message arrives as 0.9^20 ≈ 0.1 — almost nothing; the last runner has nothing to act on. If they pass 2s it arrives as an explosion. Deep training needs legs that neither shrink nor swell the signal — that's not a slogan, it's a variance bookkeeping exercise, done in the next two lessons.
Signal magnitude (log scale) across 12 layers — forward (activations, top) and backward (gradients, bottom).
per-layer gain: 0.30
Per-layer factor 0.30 < 1: every layer thins the signal. Drop into the forward bars first and the backward gradient dies completely — early layers effectively stop training. Sigmoid is worst; a scale-tune below ~1 with tanh just dies slower.
Classic variance bookkeeping (Glorot 2010 / He 2015): each layer multiplies the signal by one factor. Rising bars = trajectory healthy. Illustrative: uniform layers of a fixed width — real nets have correlations and non-linearities’ tails; the headline shapes (sigmoid sink, tanh slow leak, ReLU all-or-nothing) are faithfully shown.
What the artifact is doing
The forward signal (activations) and the backward signal (gradient) both traverse the same layers. The bars show their magnitude at each layer, for three activation choices and a scale knob standing in for initialization:
- Sigmoid-family: multiplies by < 1 every step, both directions → the sunken-gradient anatomy of the 1980s–2000s deep net.
- ReLU-family: doesn't automatically shrink, but zeroes exactly half its inputs — dead units and thin flow.
- Scale knob: the init-scale multiplier is where you, the initializer, come in — too small starves the signal, too large detonates it. Goldilocks zone stability is the actual product of chapter 2.
It's a bookkeeping problem, not a mystery problem
Notice that the math above is not mysterious; it's arithmetic. Nobody disputed that deep nets could express something great — the obstacle was that the learning signal was arithmetically guaranteed to degenerate on the way down. Every fix in this chapter is a way of keeping the product's intermediates near 1:
| Fix (coming up) | Keeps what near 1 |
|---|---|
| Init | activation variance entering each layer |
| Normalization | activation statistics passing between layers |
| Residual paths | gradient pathways (an additive shortcut) |
| Careful optimizer | step scale, net-of-all-layers |
Illustrative vs real
Variance propagation here follows the classic setup (Glorot/Bengio 2010, He et al. 2015) analytically — layers of a fixed width on independent-normal activations. Real nets have correlations, biases, and layer-type structure; their flow plots are noisier, but every headline effect (sigmoid sink, ReLU half-zeroing, init scale's win/lose switch) survives the real-data version.
Where next: init-and-norm — the two knobs whose entire job is keeping those bars level, and why they took the field 20 years to converge on.
This lesson has exercises attached — predicting gradient decay from activation/init choices — launching once the exercises layer ships.