Residual connections
In 2015 depth quietly stopped working — and the fix was embarrassingly simple: let each layer output the identity plus a correction. ResNets went 100+ layers deep.
Through 2014, stacking layers paid off until about ~20 layers — and then worse with more depth, not because of overfitting (training loss degraded too) — the deep plain networks couldn't even learn their shallower counterparts' behavior. The 2015 fix (He et al., ResNet) re-wrote the layer as:
instead of .
Learn the correction, not the signal
The identity path is free: whatever comes in, goes through. Each conv block only learns what to add — how this layer's input should be corrected. Learning "output = unchanged" is trivially easy (F→0); learning "output = a totally new copy of the input, built painstakingly from it" is the nightmare the plain stack asked. With residuals, the default passing answer is identity, and layers spend their capacity on meaningful corrections.
Gradient strength arriving at each layer (L1 deepest) for a plain stack vs a residual one, same per-layer correction gain.
Plain: by layer 17 the arriving gradient is under 2% of the top — early layers effectively untrained. Residual stack (additive identity path): every layer still receives an order-of-magnitude workable signal.
Analytic variance bookkeeping: plain = factor^depth, residual = identity + attenuating correction (additive path survives any factor). Mirrors He et al. 2015; real per-layer flow is noisier, but “plain dies, residual flows past 100 layers” replicates everywhere.
Why the gradient suddenly survives
Backward, the same shape appears: with residuals, each layer's gradient is
— identity plus a correction, rather than a bare multiplicative factor. Multiply a hundred of those terms and the product doesn't necessarily shrink toward zero; there's always a clear additive path carrying the gradient home. The gradient-flow lesson's bookkeeping (keep factors near 1) had its strongest weapon delivered by architecture instead of precision tuning.
It went deep, immediately
ResNet-152 beat everything in 2015; variants train past a thousand layers today. And the idea hopped domains fast: every transformer — your LLM book's architecture included — is stacks of residual blocks. The residual formulation turns out to be the right default grammar for deep stacks, densely connected variants included (DenseNet adds to a growing concat).
Residual ≠ dropout ≠ norm — but they interlock
Norm layers (chapter 2) condition inside the block; residuals restructure the path itself. The one-two punch — norm within, skip across — is what modern architectures share everywhere. One practical caveat learned at scale: with residuals in place, removing norm re-binds the exploding behavior; the two work best together.
Illustrative vs real
Gradient magnitudes shown use the same analytic variance-bookkeeping as the gradient-flow artifact, with the residual path modeled as an added identity (the additive term dominating — this is the honest distillation of why depth in ResNets stops decaying). Real nets' per-layer flow is noisier but the headline (plain dies, residual flows) is the replication-stable truth.
Where next: convolution-cost — convs are cheap to run; knowing where the parameters and compute land is what makes designs comparable.
This lesson has exercises attached — predicting where accuracy breaks without skips — launching once the exercises layer ships.