Init and norm
Two knobs, one job: keep activation statistics steady as the signal crosses layer after layer. This fix collectively took the field twenty years.
The previous lesson ended in bookkeeping: keep the per-layer factors near 1 and the signal survives. There are two places to touch the network to do that — what the weights are before training starts (initialization), and what happens between layers during the forward pass (normalization). Old-school researchers spent two decades conflating them; both matter, and they do different halves of the job.
Initialization: the variance handoff
Before any data arrives, weights are random — but from what distribution? The Xavier/Glorot and He analyses treat it as variance conservation:
For this to stay constant you want — i.e. wider layers want smaller weights, in exact proportion to their input count. Ignore it and you get the previous lesson's two failure modes at initialization: too-collapsed (forward signal dies inside a few layers) or too-hot (everything saturates).
Normalization: re-centering mid-stream
Init fixes layer entry conditions once. But during training, the weights feeding any layer keep moving — every update of every earlier layer changes what a layer receives. This individual-layer viewpoint is the idea Ioffe & Szegedy called internal covariate shift, and their 2015 fix is now universal machinery:
- Before each nonlinear layer, standardize the activations of the batch: subtract their mean, divide by their std. Then restore an adjustable scale and shift (, ) that the network learns:
Why this is so much of the magic
Every layer gets a fixed, normal-scale input distribution no matter what the other layers do. Learning becomes far better-conditioned — a step-up an earlier layer takes doesn't undo the step this layer was mid-way through. Modern variants batch-relative or architecture-relative (LayerNorm — what every transformer uses, including in the LLM book's positions and normalization lesson) — the principle is constant: keep statistics steady, let the learn what shape "special" actually wants.
What one layer *receives*, as the layers below it learn and shift things under it — with and without normalization.
Early training: stats roughly sane, no drift. The right histogram is overkill — until the lower layers start moving.
Illustrative: Gaussian activations with a synthetic drift knob for what other layers do mid-training (batch statistics are real, batch sizes introduce per-batch noise not shown; LayerNorm/BatchNorm swap per architecture — the *recentering* is the invariant idea).
Division of labor
- Init sets the handoff statistics right at the start, so training begins in a regime where the network's forward pass isn't soup. It's a one-time decision per architecture.
- Norm maintains the regime throughout training, as weights wander. It's continuous insurance, and secondarily a mild regularizer (batch statistics introduce a bit of noise per batch).
Miss either and the other can't compensate: a hand-tuned norm can't rescue an exploding init at scale, and a perfect init decays under a few thousand updates without norm sitting on top.
Illustrative vs real
The artifact uses Gaussian activations (the analytic case the init theorems assume) with a synthetic drift knob for what earlier layers do during training. Real distributions inside real nets are messier — occasionally multi-modal, correlated between channels — which is exactly why batch-vs-layer-vs-group norm variants coexist; the re-centering effect shown is real wherever the machinery appears.
Where next: dropout — the other insurance policy: not steadier signal, but stronger structure.
This lesson has exercises attached — matching variance-conservation scales to layer widths — launching once the exercises layer ships.