Universal approximation
A deep-enough ReLU net can approximate anything continuous. So why is that theorem famously unhelpful? Because it says nothing about finding the weights.
The most-quoted theorem in deep learning deserves its own lesson, so its limitations aren't a footnote. In plain language, the universal approximation theorem says:
What that promise actually is
Somewhere in the space of possible weight configurations there exists a network that can approximate your target function arbitrarily well. That's a statement about reach, not about how to get there. The theorem is existential — it hands you a mathematical room where success lives, with no map of the room and no promise that gradient descent, the only tool you actually have, will walk into it.
Noisy 1-D target (dashed) — your network, trained live with vanilla SGD.
no run yet
Press train a few times with the same settings and watch the final loss swing run to run — the theorem guarantees *some* weights this good exist; nothing here guarantees which weights SGD lands on. Few units underfit the wiggle; more units fit it more closely — but notice the more-compact fits also start following individual noise dots.
Real gradient descent (tiny scale): a 1-D ReLU network trained in your browser — no shortcuts. Shows the honest version of universal approximation: with enough units the *capacity* is there, and whether train-up reaches it is a separate question the theorem never answers.
The fine print that matters
- Existence ≥ finding. The theorem has a non-constructive proof. Nothing about it — and this is the gap two generations of students fall into — guarantees your training run ends near a good network.
- "Continuous" and "compact." The promise is on bounded input ranges and doesn't include discontinuous targets; real learned functions want interpolation anyway, not memorization of arbitrary corners.
- "Approximate" ≠ "extrapolate". The theorem promises nothing outside the region you specified. Network behavior outside training data is governed by nothing except whatever the optimizer happened to settle at.
- Single hidden layer suffices — in the last lesson's terms, width beats depth expression-wise. Depth won in the earlier lesson because depth wins efficiency (accuracy per parameter) and trainability, not because layer-1-only nets can't express the same functions in principle.
So is the theorem useless?
It's the floor of the field, and the honest reading is exactly the reverse of the hype reading: it demonstrates a problem rich enough to need deep optimization — the subject of chapter 2. Existence proves the goal isn't absurd; proving it reachable is what modern training practice (init, norm, optimizers, schedules) does, and the in-theory-unsolved success of SGD remains one of the great open questions of the field.
The gradient descent lesson in the ML book already warned you: descent is local. Depth makes the landscape enormous — and the next lesson makes it visible.
Illustrative vs real
The artifact trains a real 1-D ReLU network in front of you — vanilla SGD, small fixed budget — because that gap between "the theorem says it exists" and "what SGD actually finds" is the whole story, and it's more honest to show the gap live than to manufacture a picture where SGD always wins.
Where next: the-loss-landscape — what the space training actually walks through looks like, up close.
This lesson has exercises attached — deciding randomly-initialized vs "the theorem would find it" — launching once the exercises layer ships.