Regularization
Deliberately handicapping your model can make it better. Here is the dial that builds the right bias on purpose.
Chapter 1's degree-12 polynomial got sick exactly one way: its coefficients exploded. Regularization attacks that directly — add the size of the weights to the loss, and let the optimizer balance fitting the data against staying small.
The deal, stated plainly
You are not trying for the best fit of the training data — you're trying for the best fit that can be described simply. Simpler models transfer; complicated ones memorize. The penalty term prices complication.
Two features: a real signal, and pure noise made loud — the penalty decides what gets listened to
w (signal) 2.201 · w (noise) -0.174 · train MSE 12.85
signal weight
noise weight
noise instructing the model — overfit
read the deal
λ=0: nothing loud is shrunk — the noise feature *earns weight it doesn’t deserve*, train MSE dips, new-data error doesn’t. λ grows: both weights slide, noise hits zero first (it only ever fit accident), signal keeps its slope — smoother model, slightly higher train MSE, better new-data score.
You are buying predictability with fit: coefficients shrink toward simple, honest behavior, and the model stops telling you stories it read in noise. Same economics drive weight decay in the LLM book — the number never learns it was a penalty.
The two penalties, and why people fight about the difference
With loss L, fit L + λ·penalty(weights) where:
- L2 (ridge): penalty
Σ w². Shrinks all weights smoothly toward small-but-nonzero — keeps every feature, dials each down. Derivable, differentiable, stable default. - L1 (lasso): penalty
Σ|w|. Pushes small weights exactly to zero — it performs feature selection at training time: irrelevant columns drop out of the model, not out of relevance.
Same data, both fights: strong L2 says "every feature deserves a fair small piece"; strong L1 says "few features, pick them for me." Which you want depends on what you'll do with the model — interpret which factors matter (L1's sparsity is a report) or squeeze best accuracy (L2's smoothness).
The bigger lesson: bias as an investment
Nothing about a penalty knows your data — it is a prior belief ("the truth is probably smooth, coefficients are probably modest") expressed mathematically and tuned with cross-validation. You trade training-loss you can't trust anyway, for stability you can. And it generalizes far beyond classical ML:
- Polynomial of degree 12 with small coefficients ≈ a mild curve — the overfitting leash held by the numbers themselves rather than the degree dial.
- Transformer checkpoints: weight decay (L2), dropout (randomly freezing weights), early stopping — three spellings of the same trade, and all of them make training error worse while making the model better. Chapter 1's U-curve, met and paid, at industrial scale.
Illustrative vs real
A tiny 1-D dataset, one weight knob, and standardized features. Real regularization strength is tuned via CV (chapter 3), often per-feature scaled so penalties are comparable, and neural nets mix several regularizers at once.
Where next: chapter 4 — bagging and forests — where many overfit trees, averaged, become each other's regularizer.