Overfitting, bias, and variance
A model that is perfect on the data you gave it is often useless on data you didn't. Why — and what to do about it.
Here's the trap the whole book pivots on. Take training loss — the number gradient descent minimizes — and drive it to zero. Job done?
No. The loss you can touch is a proxy. The loss you actually care about is on new data the model has never seen — and those two numbers can move in opposite directions.
The party-trick model
A model with too much freedom doesn't learn the pattern; it memorizes the sample — quirks, noise, typos and all — like a student who memorized the answer key instead of the subject. Aced the practice exam, fails the real one.
11 training points, one flexible dial — two errors
train error 0.29 · new-data error 0.30
Degree 1 is too rigid (underfit): it can’t hold the hump, so it’s wrong *everywhere*. Crank to 11-12 and the curve threads every point using the noise — training error ≈ 0, new-data error explodes (notice the wild swings near the edges). The sweet spot sits around 3-5, where the model holds the pattern without the noise. Coefficient magnitudes (modest) are a visible symptom of memorization, which regularization (chapter 3) directly attacks.
Infall: the shape of the two curves
The artifact replays the ugliest chart in machine learning:
- Fitting noise rather than signal needs a flexible model — literally, a wigglier curve. Flexibility sits on a dial: degree 1 is a rigid straight line, degree 12 can thread through every point.
- The training curve only trends down — more flexibility always fits the sample at least as well.
- The new-data curve is U-shaped: too rigid, and the model can't represent the real pattern (underfitting); too flexible, and the memorized noise is wrong elsewhere (overfitting). Between them is the sweet spot — and the whole practice of ML is walking toward it honestly.
Bias is the rigid end: error from the model family being too simple to hold the truth. Variance is the flexible end: the model contorts to match the specific sample, so resampling the data would give a noticeably different model. They pull against each other; capacity is the tension between them.
The honest rule that follows
You can't pick the sweet spot by looking at training loss — it always says "more." The only legitimate umpire is held-out data. That is why chapter 3 (train/test splits, cross-validation) exists, and it's why "regularization" — deliberately handicapping a model — can massively improve it: a penalty on flexibility is a bias you choose on purpose, to buy a discount on variance.
This idea never leaves you
Transformer training in the LLM book: dropout (freeze random pieces), weight decay, early stopping — all are regularization, all of them trade training loss for generalization. Same disease, bigger patient.
Illustrative vs real
The artifact's new-data "true pattern" is a smooth curve plus known noise — clean enough for the U-shape to be visible. Real test sets are scarier: they reveal failure modes you didn't think to plot.
Where next: chapter 2 gives better function families — starting with linear-regression. Cross-validation in train-test-splits is the honest umpire.