Bagging and random forests
Train many proud, faulty models on different slices of the data and let them vote. Their errors cancel; their brains accumulate.
One decision tree, fully grown, overfits hard — chapter 2 said so. The random forest's bet is that you don't have to fix the tree; you fix the crowd. Fit many full-grown, independent noisy trees and average their answers. Each tree memorizes its sample's quirks in a different direction — and those directions cancel, while the genuine pattern, which every tree rediscovers, accumulates.
Why it works — the variance math
Averaging k independent predictors of the same thing divides the variance by k. Individually each tree is a shaky witness; the jury is the payoff. Bias stays, variance collapses — chapter 1's trade, paid in compute.
Committee of 7 trees, one question at a time — majority decides
majority: no · correct answer: no · committee right
each tree saw a different feature slice, so quirks differ and cancel
Flip through questions with the lottery on: majorities hold despite individual dissent. Turn it off and the committee degenerates to one voice wearing seven hats. That failure mode is why “random” sits in the name — independence is not a nice-to-have in ensembling; it IS the mechanism. Note what’s packaged with each no-vote: a mislabeled *bias* in some trees never leaves — averaging kills variance, not bias.
The two sources of independence — both cheat, on purpose
- Bootstrap samples (the bagging): each tree trains on a random sample with replacement — same size as the original, so ~37% of rows repeat and another ~37% never appear (that leftover slice, "out-of-bag", is a free validation set — an honest score with zero extra data).
- Random feature subsets: every split considers only a random handful of the features (sqrt count for classification). Without this, one wildly predictive feature (age 60 → fraud) puts itself at the head of every tree and the crowd votes in unison — averaging correlated estimators saves nothing. The feature lottery is what keeps the votes independent.
When it wins — and the cost you pay for it
Forests dominate tabular data in practice: hard to overfit catastrophically (adding trees never hurts), need little tuning, tolerate messy features, and no sum reads off more than "these columns matter." Left unpriced: you lose the single tree's interpretability — a forest of 500 trees is nobody's audit trail — and prediction cost grows linearly with trees. That cost buys near-SOTA tabular accuracy without a gradient descent loop anywhere in sight, which, given everything in chapters 1-3, should strike you as remarkable.
Illustrative vs real
12 voters on a toy committee here. Real forests run 100-1000 trees, and the surprisingly stable result they enjoy comes from the same two-part lottery — bootstrap rows, restricted features, every time.
Where next: boosting — a different crowd: models that correct each other in order.