Gradient descent
A slope, a step size, and thousands of downhill steps — the one optimization rule that trained almost everything you've heard of.
Sliders don't scale. A modern model has millions to billions of parameters — there is no hand (and no grid search) for that space. The procedure that replaces you is barely an idea:
Stand somewhere on the loss landscape. Feel which way is downhill. Take one small step that way. Repeat until you stop descending.
Why it works for any model
Calculus gives a universal answer to "which way is downhill?" — the gradient (the partial derivative of the loss with respect to each parameter). Computing gradients + nudging parameters is the entirety of backpropagation from the LLM book's training chapter; every neural network in existence is fit by some version of this loop.
Two starting points, one rule: step against the gradient
pick a start; watch the trail
One rule — descend opposite the slope — from any start. Push learning rate past ~0.3 and watch the ball overshoot valley walls and oscillate or explode; that asymmetry is why humans shrink the rate over training. Real landscapes aren’t hand-drawn, the geometry is.
The three things that decide whether it works
- Learning rate (the step size). Too small: training crawls, hours become days. Too big: each step overshoots the valley, and the loss oscillates or explodes instead of settling. The default practical recipe is "too big at first, then shrink" — a learning-rate schedule.
- Batching. Computing the gradient using all the data per step is exact but slow; using one random example is fast but noisy. "Mini-batch" (hundreds of examples) is the compromise everyone uses — and that noise turns out to be a feature: it rattles the descent out of small local dents.
- Where you start. Descent is local: it flows into the nearest valley, and can't see over ridges. Deep nets dodge this — in high dimensions there is almost always some nearby downhill route, which is a large part of why giant models train at all. Classical small models (trees aside) get convex, bowl-shaped losses where any start finds the same bottom.
Local minimum fears are mostly outdated
The classic worry "what if it gets stuck in a bad local minimum?" mattered for small hand-crafted networks. In millions of dimensions, saddle points — flat spots where some directions go up and others go down — are the actual obstacle, and momentum/adaptive optimizers (SGD+momentum, Adam) blast straight through them.
And that's the engine
Write the loop once — compute loss, compute gradient, parameters -= learning-rate × gradient — and you have stochastic gradient descent,
the common ancestor of model training in this book and of GPT training in
the LLM one. Everything classical adds is a smarter family of functions
(linear, logistic, trees) and a better sense of what "the minimum" should
even mean on data the model hasn't seen — which is the next lesson's
subject, and the single most important idea in the whole book.
Illustrative vs real
The artifact shows descent on a 2-D landscape so the geometry is visible. Real gradients live where you can't draw them — but the rule "step opposite the gradient" is dimension-free, which is the quiet miracle that makes all of this work.
Where next: overfitting-bias-variance — the loss you can drive to zero on training data may not be the loss you care about.