Wiki
Core7 min read

Bagging and random forests

Train many proud, faulty models on different slices of the data and let them vote. Their errors cancel; their brains accumulate.

One decision tree, fully grown, overfits hard — chapter 2 said so. The random forest's bet is that you don't have to fix the tree; you fix the crowd. Fit many full-grown, independent noisy trees and average their answers. Each tree memorizes its sample's quirks in a different direction — and those directions cancel, while the genuine pattern, which every tree rediscovers, accumulates.

Why it works — the variance math

Averaging k independent predictors of the same thing divides the variance by k. Individually each tree is a shaky witness; the jury is the payoff. Bias stays, variance collapses — chapter 1's trade, paid in compute.

Committee of 7 trees, one question at a time — majority decides

tree #1
yes
tree #2
no
tree #3
no
tree #4
yes
tree #5
no
tree #6
yes
tree #7
no

majority: no · correct answer: no · committee right

each tree saw a different feature slice, so quirks differ and cancel

Flip through questions with the lottery on: majorities hold despite individual dissent. Turn it off and the committee degenerates to one voice wearing seven hats. That failure mode is why “random” sits in the name — independence is not a nice-to-have in ensembling; it IS the mechanism. Note what’s packaged with each no-vote: a mislabeled *bias* in some trees never leaves — averaging kills variance, not bias.

The two sources of independence — both cheat, on purpose

  1. Bootstrap samples (the bagging): each tree trains on a random sample with replacement — same size as the original, so ~37% of rows repeat and another ~37% never appear (that leftover slice, "out-of-bag", is a free validation set — an honest score with zero extra data).
  2. Random feature subsets: every split considers only a random handful of the features (sqrt count for classification). Without this, one wildly predictive feature (age 60 → fraud) puts itself at the head of every tree and the crowd votes in unison — averaging correlated estimators saves nothing. The feature lottery is what keeps the votes independent.

When it wins — and the cost you pay for it

Forests dominate tabular data in practice: hard to overfit catastrophically (adding trees never hurts), need little tuning, tolerate messy features, and no sum reads off more than "these columns matter." Left unpriced: you lose the single tree's interpretability — a forest of 500 trees is nobody's audit trail — and prediction cost grows linearly with trees. That cost buys near-SOTA tabular accuracy without a gradient descent loop anywhere in sight, which, given everything in chapters 1-3, should strike you as remarkable.

Illustrative vs real

12 voters on a toy committee here. Real forests run 100-1000 trees, and the surprisingly stable result they enjoy comes from the same two-part lottery — bootstrap rows, restricted features, every time.

Check yourself

Eduspheria wiki · Classical Machine Learning, Ensembles

0 / 5 answered

  1. 1Averaging k independent predictors divides which quantity by k?
    Multiple choice
  2. 2What is the sampling scheme that gives each tree a same-size random sample drawn with replacement?
    Short answer
  3. 3A bootstrap sample of size n contains roughly what percentage of the distinct original rows? Give the whole number.
    Numeric answer
  4. 4Without random feature subsets, one dominant feature can make every tree vote alike and defeat the averaging.
    True / false
  5. 5What does randomizing the feature subset at each split protect the forest against?
    Multiple choice

Where next: boosting — a different crowd: models that correct each other in order.