Boosting
Invert bagging's geometry: train models *in sequence*, each one obsessed with what the previous ones got wrong.
Bagging averages independent witnesses. Boosting composes dependent, sequential specialists. Train a first weak model — usually a very shallow tree ("decision stump"). Find where it's wrong. Train a second model focused on those failures. Repeat, in order, hundreds of times. The final answer is the weighted sum of all of them.
Like adding digits of precision
Each round adds a small correction term to the running answer — closer to truth, piece by piece. It's not a vote; it's an accumulation.
Stumps correcting residuals, one round at a time
avg |residual|: 0.188
Orange stubs are the residuals the next stump will chase — they shrink every round. Each added step is a tiny depth-limited tree; the running sum is the model. Chasing too long overfits on noise exactly like the last three chapters warned — the stubs would happily fit the scatter itself. Real libraries add a learning rate ≤ 0.1, early stopping on a validation fold, and stop at round 2000 — not because the loop changed, but because the discipline got automated.
Three design decisions, all visible up there
- Weakness is the design, not the flaw. A depth-1 tree is nearly useless alone — on purpose. Each specialist needs limited freedom so it can only craft a correction, never a takeover. It's chapter 3's regularization reappearing at ensemble scale: keep each piece small.
- Two spellings of "focus on the failures." Classic AdaBoost reweights training rows: misclassified cases get heavier, so the next learner spends its capacity there. Gradient boosting instead fits residuals — the part of the answer not yet explained. Same instinct, different bookkeeping.
- The sequence is the point. Round n's training set exists because round n−1's residuals did. No shuffling, no votes — every specialist inherits and rewrites a running answer.
And the trap, which is inherited too
Sequential dependence compounds errors: nothing in the loop knows when to stop chasing noise. Training error keeps dropping well past the point where validation error does — every chapter-3 lesson recurs here, because the loop is more eager than plain gradient descent, not less. Production libraries (XGBoost family) carry: a small learning rate (one small correction per round), depth limits per stump, and automatic early stopping on a validation fold. Willingly overfits in under 100 rounds unwatched; wins after 2000 watched.
Illustrative vs real
The artifact corrects a 1-D fit in visible steps with a hand-tuned ladder of stumps standing in for real boosting rounds. Modern gradient-boosting from XGBoost/LightGBM is that same loop, tuned to hardware — handle missing values and categorical splits natively, and parallelize where it can.
Where next: gradient boosting — renaming what you now know: it is gradient descent, run in function space.