The Wiki
Core8 min read

Optimizers

Gradient descent is the rule. Momentum, adaptive scaling, or both — three physical upgrades that decide whether training takes hours or days.

The ML book's gradient descent gave you the raw rule: step opposite the gradient. Chapter 1's loss-landscape lesson showed where that rule suffers — ravines make plain descent zigzag across a steep valley instead of gliding along it. Every modern optimizer is a physical tweak added to the raw rule to fix one of those behaviors. There are exactly three:

1. Momentum: give descent a direction memory

vt+1=βvt+gt,wt+1=wt−η vt+1 v_{t+1} = \beta v_t + g_t, \qquad w_{t+1} = w_t - \eta\, v_{t+1}

A rolling ball. Velocity accumulates past gradients: signs that keep agreeing get amplified (the valley floor direction — few agreeing steps in a row there), directions that keep flipping (the steep walls, zigzag cancelling) average toward zero. Small β\beta (~0.9 is standard.

2. Per-parameter scaling: Adam's idea

The raw rule treats every parameter identically, even though the gradient magnitude per parameter differs by orders of magnitude. Adam keeps running first- and second-moment estimates (mt,vtm_t, v_t) of each parameter's own gradient and normalizes:

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2,wt+1=wt−η mtvt+ϵ m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2, \quad w_{t+1} = w_t - \eta\, \frac{m_t}{\sqrt{v_t} + \epsilon}

Each parameter takes a step sized by the reliability of its own signal, not its raw loudness. Sparse or quiet parameters get louder effective steps; consistently-hot ones get damped. Adam ≈ momentum + per-parameter conditioning, and it's the default worldwide for a reason.

3. Weight decay: keep it small on principle

A mild constant pull toward zero, w←w−ηλww \leftarrow w - \eta\lambda w, added to every step — independent of the gradient. It's the ridge regularization the ML book taught, repurposed: small weights are harder weights to memorize with. In modern implementations it's set per-group (no decay on norm layers, bias, and such), detached from the optimizer proper.

Same start, same ravine (long valley, steep walls), same learning rate — three real optimizers race.

— SGD crosses the valley wall-to-wall: every step is exact and over-corrects across the steep axis while creeping along the floor.

— Momentum: wall-crossings average out, valley-direction agreement accumulates — it glides.

— Adam rescales each axis by its own gradient history: steep axis damped, flat axis amplified.

Full-batch arithmetic here; real mini-batch noise further rattles SGD — helping it escape dents and stalling it otherwise. Same start, same η: the difference is entirely in what each method remembers.

Real update rules (β=0.9 momentum; β₁=0.9, β₂=0.999 for Adam) integrating the same ravine: ∇L = (x/12, y). Textbook ravine shape — the zigzag-vs-glide physics transfers to every real valley in every dimension.

What to actually use

Without horror stories behind them:

  • AdamW — Adam with the decoupled weight decay above — for the overwhelming majority of cases, including all transformer training (your LLM book's efficiency lesson lives on it).
  • SGD + momentum + schedule — still the choice for classical convnets where Adam often generalizes slightly worse; never a "dead" option worth burying.

Adam is not universally better

Intuition suggests Adam should always win and it often does in speed. But the long-run generalization story — which checkpoint actually tests better — sometimes favors plain SGD's heavier final wandering. When in doubt: AdamW for anything transformer-like; SGD+momentum for CNNs; diagnostics before dogma.

Illustrative vs real

The artifact races all three for real — actual gradient descent arithmetic on a fixed 2-D ravine, plotted from same start. Missing in 2-D: the batch noise that rattles real runs (here descent is full) and saddle-plateaus (the 2-D slice's floor is honest but simplified). Zigzag vs glide is nevertheless the exact physics everywhere, in any dimension.

Where next: learning-rate — every optimizer above still bows to one single knob.

This lesson has exercises attached — picking the optimizer family for a described task — launching once the exercises layer ships.