Gradient boosting
The boost loop and gradient descent are the same algorithm in two costumes — one on numbers, one on whole functions.
Chapter 1's descent rule: parameter -= rate × gradient. This chapter's boosting loop: model_n = model_{n-1} + learning-rate × tree_fitted_to_the_residual.
They have the same shape, and that rewriting is not a metaphor: gradient
boosting is gradient descent where the "parameters" are functions.
Function space, one sentence
Instead of nudging numbers inside one fixed function, add small functions to a running sum of functions. Each new tree is one step, taken in the direction that reduces the loss most — which is exactly what the negative gradient says. Everything else is plumbing.
Function-space descent: each round adds one small tree
mean |error| after 0 rounds: 0.188
Round “step” = one tiny tree fit to what’s left unexplained, added at learning-rate strength. Crank learning rate to 0.8+: early rounds overshoot into zig-zag (chapter 1’s exploding step size, reincarnated). Small rate, more rounds: same destination, calmer road. This IS gradient descent — the “parameter” being stepped is a whole function.
The descent, walked step by step
- Start with a constant prediction (mean target).
- Compute residuals — for squared loss, just
truth − current model(the artifact's orange stubs). - Fit a small tree to the residuals (not the truth).
- Add that tree's prediction × learning-rate into the running model.
- Repeat thousands of times with a learning rate in the 0.01–0.1 band.
Reading it back, the residual tree is the negative gradient in function form (for squared loss it's literal; for other losses the "pseudo-residual" generalizes, which is how the same engine does classification and ranking too). The learning-rate dial is chapter 1's lesson playing its familiar role: small steps, integral corrections, untold overfit avoided. XGBoost's celebrated "shrinkage" is a chapter-1 learning-rate schedule; its row/column subsampling per boost round is bagging, inside boosting, on purpose.
What to file this under
File it as the LLM era's bridge lesson: neural networks descend in parameter space (chapter 1), gradient-boosted trees descend in function space (here), and both live and die by the same chapter-3 disciplines — validation, early stopping, regularization. When XGBoost still beats a neural net on tabular problems (it frequently does), this is why: it is neural-style training. Simply another training algorithm on another representational bet.
Illustrative vs real
The artifact's stumps are "quantized truth with 2^rounds bands" — standing in for each round's residual-fitting stump, which for squared loss is exactly what a depth-limited tree produces early on. Real XGBoost adds second-order derivatives (Newton-like steps), histogram splits, and shrink — strictly refinements of the loop above.
Where next: chapter 5 loosens supervision — clustering asks the model to structure data with no labels at all.