Wiki
Advanced8 min read

Gradient boosting

The boost loop and gradient descent are the same algorithm in two costumes — one on numbers, one on whole functions.

Chapter 1's descent rule: parameter -= rate × gradient. This chapter's boosting loop: model_n = model_{n-1} + learning-rate × tree_fitted_to_the_residual. They have the same shape, and that rewriting is not a metaphor: gradient boosting is gradient descent where the "parameters" are functions.

Function space, one sentence

Instead of nudging numbers inside one fixed function, add small functions to a running sum of functions. Each new tree is one step, taken in the direction that reduces the loss most — which is exactly what the negative gradient says. Everything else is plumbing.

Function-space descent: each round adds one small tree

mean |error| after 0 rounds: 0.188

Round “step” = one tiny tree fit to what’s left unexplained, added at learning-rate strength. Crank learning rate to 0.8+: early rounds overshoot into zig-zag (chapter 1’s exploding step size, reincarnated). Small rate, more rounds: same destination, calmer road. This IS gradient descent — the “parameter” being stepped is a whole function.

The descent, walked step by step

  1. Start with a constant prediction (mean target).
  2. Compute residuals — for squared loss, just truth − current model (the artifact's orange stubs).
  3. Fit a small tree to the residuals (not the truth).
  4. Add that tree's prediction × learning-rate into the running model.
  5. Repeat thousands of times with a learning rate in the 0.01–0.1 band.

Reading it back, the residual tree is the negative gradient in function form (for squared loss it's literal; for other losses the "pseudo-residual" generalizes, which is how the same engine does classification and ranking too). The learning-rate dial is chapter 1's lesson playing its familiar role: small steps, integral corrections, untold overfit avoided. XGBoost's celebrated "shrinkage" is a chapter-1 learning-rate schedule; its row/column subsampling per boost round is bagging, inside boosting, on purpose.

What to file this under

File it as the LLM era's bridge lesson: neural networks descend in parameter space (chapter 1), gradient-boosted trees descend in function space (here), and both live and die by the same chapter-3 disciplines — validation, early stopping, regularization. When XGBoost still beats a neural net on tabular problems (it frequently does), this is why: it is neural-style training. Simply another training algorithm on another representational bet.

Illustrative vs real

The artifact's stumps are "quantized truth with 2^rounds bands" — standing in for each round's residual-fitting stump, which for squared loss is exactly what a depth-limited tree produces early on. Real XGBoost adds second-order derivatives (Newton-like steps), histogram splits, and shrink — strictly refinements of the loop above.

Check yourself

Eduspheria wiki · Classical Machine Learning, Ensembles

0 / 5 answered

  1. 1Gradient boosting is best described as gradient descent in what space?
    Multiple choice
  2. 2What does each new tree in gradient boosting fit?
    Short answer
  3. 3The running model predicts 10 and the true target is 13. For squared loss, what is the residual truth minus prediction?
    Numeric answer
  4. 4XGBoost's shrinkage is essentially a learning rate applied to each boosting step.
    True / false
  5. 5On tabular data, gradient-boosted trees often still beat neural networks because:
    Multiple choice

Where next: chapter 5 loosens supervision — clustering asks the model to structure data with no labels at all.