Wiki
Core7 min read

Dropout

Deliberately break parts of the network on every step. It sounds like sabotage — it's the cheapest generalization tool we have.

Overfitting from the ML book (bias–variance lesson) is the deep net's home disease: giant capacity, data-hungry, fully capable of memorizing its training set wholesale. Dropout (Srivastava et al., 2014) hits it with an odd weapon: on every training step, delete a random pp fraction of each layer's units — edges, activations, everything — train on what's left, and re-roll the dice next step.

at train time (per step, per unit):a′={awith probability 1−p0with probability p \text{at train time (per step, per unit):}\quad a' = \begin{cases} a & \text{with probability } 1-p\\ 0 & \text{with probability } p \end{cases}

No unit can count on any particular partner. Each step trains a different thin network — a unit can never rely on the same partners twice in a row, so collusiveness is inhibited. At test time, everything is on, scaled by 1−p1-p to keep the mean right.

Why forced redundancy generalizes

Two mechanisms, both honest:

  1. No co-dependence. Units that can rely on a specific neighbor can learn gimmicks shared only with that neighbor (a textbook memorization pattern). With partners vanishing randomly, each unit must work broadly — a feature, not a particular pattern.
  2. Ensembling for free. Each step trains a different subnet, and they all share weights. Averaging an ensemble of individually-jittery predictors is far more stable than any one of them — the artifact below shows exactly that, live.

A network that has learned to split the red class above from the blue below — trained with dropout at the chosen rate.

alive experts: 2/5

Light lines: the surviving experts’ individual boundaries — notice how they disagree with each other more as p rises. Bold line: their average, the *test-time* prediction. One click = one training step: different experts survive, each step trains a different sub-net; the average stays put.

Mechanism shown: sampled thinning of an ensemble whose members share weights — each fresh mask trains a different subnet, and test-time averages surviving paths (scaled by 1−p, omitted here). Illustrative: five fixed “experts” stand in for thousands of units; real subnets overlap in structured, not independent ways.

The ensemble framing is the honest big picture

Dropout is the cheapest member of a family of "train many, average one" ideas — weight-sharing ensembles (what dropout literally is), bagging in the classical book, deep ensembles (literally training several nets and averaging their predictions), and — close to modern LLMs — Mixture of Experts, where routing choses which parts of a giant net see a given token. Whether the members are sampled or routed, the claim is the same: prediction averaging over diversity beats polish of any single path.

Where dropout doesn't get used

Being retro-honest: modern transformers mostly don't dropout their core at scale — their size + data regime overfit differently, and dropout's train-time noise is expensive in throughput terms. It's still completely standard in building-blocks (attention dropout, small fine-tuning, classical-scale convolutional nets). Know it both ways: essential for mid-size nets, niche at frontier scale.

Dropout ≠ random init ≠ data augmentation

All three inject randomness, all three reduce overfitting, they sit at different points of the pipeline. Randomness at initialization (last lesson) changes which network you start with; augmentation (chapter 3) changes what data says; dropout changes the network's *internal shape during the walk.

Illustrative vs real

The artifact shows five sampled subnets as visible jittery predictions plus their average — the live mechanism. Note that actual dropout in training also scales activations at train time in the inverted formulation and interacts with normalization layers; unshown but described in the footnote. The jitter → stable average relationship is the real effect.

Check yourself

Eduspheria wiki · Deep Learning, Training deep things

0 / 4 answered

  1. 1At test time, how does standard dropout preserve the expected activation?
    Multiple choice
  2. 2Dropout's ensemble effect comes from training many different thinned subnetworks that share weights, then averaging them at test time.
    True / false
  3. 3What classical method averages many models trained on bootstrap resamples, which the lesson names alongside dropout?
    Short answer
  4. 4A layer has 200 units and a dropout rate of 0.25. On average, how many units are dropped on a training step?
    units
    Numeric answer

Where next: optimizers — your descent is fine but slow; the three physical tweaks that make it fast.

This lesson has exercises attached — choosing a dropout rate for different dataset sizes — launching once the exercises layer ships.