The Wiki
Core7 min read

The learning rate

One knob, and it dominates. Too small wastes the most expensive compute you have; too big ejects you from the basin; the schedule is everything.

The learning rate η\eta multiplies the step. Every other training decision — architecture, init, optimizer — can be reasonable or heroic; η\eta quietly outranks all of it. The honest folklore: a

  • η\eta 10× smaller than needed and your daily job of training might cost 10× the compute, at identical final quality;
  • η\eta ~10× too large and the loss explodes — steps overshoot the basin and land above where you started, on the wall.

Why it's knife-edged, not monotone

In a convex parabola, descent converges for any η\eta up to a stability limit; above that limit, steps alternate wider and wider — divergence, not just slowness. In a real landscape, what you need is "small enough to not overshoot the sharpest curvature you land in this step, big enough to make progress across thousand-step spans". Both conditions, on the same scalar. There's no perfect setting — there's a regime, and there are schedules for navigating it.

160 real gradient steps on a rippled valley — loss vs step.

final loss: 0.998

Loss early improvement but plateau: safe epochs by the second run still overweight ripple. Steady size everything; sweep the base up until it just diverges, then back off.

Real gradient steps on an honest 1-D surface (0.25(θ−2)² + 0.35·sin 3θ — ripples stand in for batch noise/curvature of real landscapes). The overshoot-and-diverge threshold is exact for this quadratic part; real nets mix that with schedule choices at scale.

The standard modern schedule

Everyone reputable does the same theater in three acts — a warm-up, a high phase, and a decay:

  1. Warm-up ramp. Begin tiny and grow. Early in training, gradients are erratic and the loss surface misrepresents the model's final basin; a large step at step 0 flings the model somewhere random. (Transformers take this to an extreme — hundreds to thousands of warmup steps — and the LLM training chapter trains at scales where a bad first phase poisons everything.)
  2. Peak. Spend the run's compute at the largest η\eta the configuration tolerates — biggest gains per second.
  3. Anneal (cosine/step). Slowly decremented so that late updates settle the weights into a wide basin rather than jiggling across it. This is where "wide flat minimum" from the landscape lesson — the one that generalizes — is actually reached.

The schedule's essence: your optimal η\eta is not a constant — it's a curve, aggressive when gradients are trustworthy, gentle when they decide where you come to rest.

How people find it

  • LR finder (Smith) — walk the LR up exponentially over a few steps, plot loss-vs-LR, pick the steepest-descent band.
  • Train short, poke the ending: train short runs with candidate rates, compare loss trajectories, extrapolate.
  • Everything else is informed defaults (transformer: 10−310^{-3}–10−410^{-4} scale with warmup; CNN: 10−210^{-2}-ish with SGD).

Illustrative vs real

An illustrative 1-D valley with a ripple, real gradient-step dynamics — the curvature-and-overshoot physics is faithful, and that's the lesson this artifact carries. Real LR picking also interacts with batch size (bigger batch = safer larger η\eta) and with warmup in ways the 1-D version can't show.

Where next: chapter 3 opens the door on specializing this machinery for images — sliding filters, an idea so successful it ran vision for ten years and still anchors its architectures.

This lesson has exercises attached — reading divergence/plateau symptoms and diagnosing the LR regime — launching once the exercises layer ships.