Dropout
Deliberately break parts of the network on every step. It sounds like sabotage — it's the cheapest generalization tool we have.
Overfitting from the ML book (bias–variance lesson) is the deep net's home disease: giant capacity, data-hungry, fully capable of memorizing its training set wholesale. Dropout (Srivastava et al., 2014) hits it with an odd weapon: on every training step, delete a random fraction of each layer's units — edges, activations, everything — train on what's left, and re-roll the dice next step.
No unit can count on any particular partner. Each step trains a different thin network — a unit can never rely on the same partners twice in a row, so collusiveness is inhibited. At test time, everything is on, scaled by to keep the mean right.
Why forced redundancy generalizes
Two mechanisms, both honest:
- No co-dependence. Units that can rely on a specific neighbor can learn gimmicks shared only with that neighbor (a textbook memorization pattern). With partners vanishing randomly, each unit must work broadly — a feature, not a particular pattern.
- Ensembling for free. Each step trains a different subnet, and they all share weights. Averaging an ensemble of individually-jittery predictors is far more stable than any one of them — the artifact below shows exactly that, live.
A network that has learned to split the red class above from the blue below — trained with dropout at the chosen rate.
alive experts: 2/5
Light lines: the surviving experts’ individual boundaries — notice how they disagree with each other more as p rises. Bold line: their average, the *test-time* prediction. One click = one training step: different experts survive, each step trains a different sub-net; the average stays put.
Mechanism shown: sampled thinning of an ensemble whose members share weights — each fresh mask trains a different subnet, and test-time averages surviving paths (scaled by 1−p, omitted here). Illustrative: five fixed “experts” stand in for thousands of units; real subnets overlap in structured, not independent ways.
The ensemble framing is the honest big picture
Dropout is the cheapest member of a family of "train many, average one" ideas — weight-sharing ensembles (what dropout literally is), bagging in the classical book, deep ensembles (literally training several nets and averaging their predictions), and — close to modern LLMs — Mixture of Experts, where routing choses which parts of a giant net see a given token. Whether the members are sampled or routed, the claim is the same: prediction averaging over diversity beats polish of any single path.
Where dropout doesn't get used
Being retro-honest: modern transformers mostly don't dropout their core at scale — their size + data regime overfit differently, and dropout's train-time noise is expensive in throughput terms. It's still completely standard in building-blocks (attention dropout, small fine-tuning, classical-scale convolutional nets). Know it both ways: essential for mid-size nets, niche at frontier scale.
Dropout ≠ random init ≠ data augmentation
All three inject randomness, all three reduce overfitting, they sit at different points of the pipeline. Randomness at initialization (last lesson) changes which network you start with; augmentation (chapter 3) changes what data says; dropout changes the network's *internal shape during the walk.
Illustrative vs real
The artifact shows five sampled subnets as visible jittery predictions plus their average — the live mechanism. Note that actual dropout in training also scales activations at train time in the inverted formulation and interacts with normalization layers; unshown but described in the footnote. The jitter → stable average relationship is the real effect.
Where next: optimizers — your descent is fine but slow; the three physical tweaks that make it fast.
This lesson has exercises attached — choosing a dropout rate for different dataset sizes — launching once the exercises layer ships.