Wiki
Core8 min read

Regularization

Deliberately handicapping your model can make it better. Here is the dial that builds the right bias on purpose.

Chapter 1's degree-12 polynomial got sick exactly one way: its coefficients exploded. Regularization attacks that directly — add the size of the weights to the loss, and let the optimizer balance fitting the data against staying small.

The deal, stated plainly

You are not trying for the best fit of the training data — you're trying for the best fit that can be described simply. Simpler models transfer; complicated ones memorize. The penalty term prices complication.

Two features: a real signal, and pure noise made loud — the penalty decides what gets listened to

w (signal) 2.201 · w (noise) -0.174 · train MSE 12.85

signal weight

noise weight

noise instructing the model — overfit

read the deal

λ=0: nothing is shrunk — the noise feature *earns weight it doesn’t deserve*, train MSE dips, new-data error doesn’t. λ grows: this is ridge (L2) — every weight shrinks smoothly toward zero, none of them exactly, so the noise weight never literally vanishes. What it buys you is still real: a smoother, less noise-chasing fit and better new-data error, at the cost of a small rise in train MSE. (A penalty that does zero weights out exactly is L1 — lasso — a different trade covered elsewhere.)

You are buying predictability with fit: coefficients shrink toward simple, honest behavior, and the model stops telling you stories it read in noise. Same economics drive weight decay in the LLM book — the number never learns it was a penalty.

The two penalties, and why people fight about the difference

With loss L, fit L + λ·penalty(weights) where:

  • L2 (ridge): penalty Σ w². Shrinks all weights smoothly toward small-but-nonzero — keeps every feature, dials each down. Derivable, differentiable, stable default.
  • L1 (lasso): penalty Σ|w|. Pushes small weights exactly to zero — it performs feature selection at training time: irrelevant columns drop out of the model, not out of relevance.

Same data, both fights: strong L2 says "every feature deserves a fair small piece"; strong L1 says "few features, pick them for me." Which you want depends on what you'll do with the model — interpret which factors matter (L1's sparsity is a report) or squeeze best accuracy (L2's smoothness).

The bigger lesson: bias as an investment

Nothing about a penalty knows your data — it is a prior belief ("the truth is probably smooth, coefficients are probably modest") expressed mathematically and tuned with cross-validation. You trade training-loss you can't trust anyway, for stability you can. And it generalizes far beyond classical ML:

  • Polynomial of degree 12 with small coefficients ≈ a mild curve — the overfitting leash held by the numbers themselves rather than the degree dial.
  • Transformer checkpoints: weight decay (L2), dropout (randomly freezing weights), early stopping — three spellings of the same trade, and all of them make training error worse while making the model better. Chapter 1's U-curve, met and paid, at industrial scale.

Illustrative vs real

A tiny 1-D dataset, one weight knob, and standardized features. Real regularization strength is tuned via CV (chapter 3), often per-feature scaled so penalties are comparable, and neural nets mix several regularizers at once.

From the exam paper

Exam-modeled · End-Sem December 2024

0 / 5 answered

  1. 1Which penalty does ridge regularization add to the loss?
    Multiple choice
  2. 2What happens to the individual coefficients under strong ridge (L2) regularization?
    Short answer
  3. 3Lasso (L1) regularization can set some coefficients exactly to zero and so performs feature selection.
    True / false
  4. 4As the regularization strength increases from zero, how do bias and variance typically change?
    Multiple choice
  5. 5Which procedure is normally used to choose the regularization strength?
    Short answer

Where next: chapter 4 — bagging and forests — where many overfit trees, averaged, become each other's regularizer.