The loss landscape
One parameter, one curve, was the ML book. Deep nets put you on a surface in millions of dimensions — here's what that surface treats as hills, basins, and roads.
With one parameter you could see the loss. With two you can still draw it. Deep networks have millions — but the shapes that matter (basins, flat stretches, ridges, ravines) all exist in 2-D slices you can look at, and the intuitions transfer surprisingly well.
What the landscape is made of
Loss isn't a property of the model. It's a property of the model on this data: run the network over the dataset with one setting of the weights, that's one loss value. Change the weights, you're standing somewhere else on the surface. Training is a walk across that surface, and surface shape determines everything: does plain descent zigzag, does it stall, where can it end up?
Click the surface to drop a gradient-descent walker (keep the last five trails).
The valley is longer than it is steep: steps bounce between the walls while creeping along the floor. That geometry is a ravine — the shape that makes plain descent zig-zag.
Ripples barely deflect a moving walker — the basin floors win. Compare trails from a wall crest vs a floor start to see who’s boss here.
Real gradient descent on a real 2-D loss surface. Local 2-D slices of deep-net landscapes behave like this locally; what 2-D can’t show is saddles in most directions at once — high-dimensional nets are rarely cornered the way 2-D walkers can be.
What the shape teaches
Drop the ball on different spots and watch where it walks:
- Ravines. Steep in one direction, shallow in another — the classic deep-net shape. Plain SGD zig-zags across the steep axis while creeping along the valley floor. This geometry — not general evil — is what momentum is for, two lessons from now.
- Ripples in the valley. Small bumps inside a big basin. SGD with a sensible learning rate ignores ripples and follows the trendline; it's the noisy mini-batch version that bounces into and out of them (and occasionally out of a tiny bad basin entirely — the ML book's batching lesson in action).
- Flat regions near the bottom. The important modern insight (Zhang et al. and follow-ups): descent doesn't find the single deepest pit; it reliably stops in wide flat basins — and wide basins are the ones that generalize to unseen data. Luck in the landscape isn't luck at all.
The dimension disclaimer
Real loss surfaces are far stranger than the slice shows: saddle points in nearly all directions at once, plateaus where "downhill" has almost no slope to measure, and interconnections between basins at scales 2-D drawings can't carry. But the intuitions — step size sensitivity, ravines, wide-vs-narrow minima — are exactly what practitioners' hands have learned. The next chapter's every trick exists because of one of these shapes.
“Where you start” stops mattering much
In the ML book, starting point decided which valley you'd find. In millions of dimensions, basins connect at high dimension, and different random starts mostly reach roughly-equivalent solutions. The start point that does matter — and dramatically — is the scale of the initial weights, which is chapter 2's opening punchline.
Illustrative vs real
A 2-D slice of a high-dimensional space, implied by decades of loss-surface papers: dimension-free geometry lessons (ravines, flat basin preference, ripple resistance) are what this artifact honestly shows. Don't carry away the exact shape as the truth.
Where next: gradient-flow — the walk down is fine, but the signal telling early layers which way is downhill has to survive the trip.
This lesson has exercises attached — reading landscape features from loss slices — launching once the exercises layer ships.