01 / 12
Begin with three observations
Suppose we want to predict these three outcomes from a single input x. Each model gives us three predictions: one for each observation.
To understand boosting, we will put those predictions into a new coordinate system.
If you came from the partition explainer, notice the change: we now study prediction space, with one coordinate per observation. This smaller example complements the main gradient-boosting story.
We treat the outcomes as numerical regression targets throughout this example.
02 / 12
One observation, one axis
The first coordinate records our prediction for observation 1. The second records our prediction for observation 2. The third records our prediction for observation 3.
These axes measure predictions. The input values x tell a model how to predict; they are not the axes of this picture.
With n training observations, this space has n coordinates.
03 / 12
The outcomes give us a destination
Place the observed outcomes at y = (1, −1, 1). A model that predicts all three outcomes exactly reaches this point.
For squared-error loss, getting closer to this target means reducing training error. The target stays in the same place throughout the story.
04 / 12
A simple tree points partway there
Start with the zero function, F₀(x) = 0. Fit a tree with just one split to the outcomes. This stump separates the first observation from the other two.
Its leaves predict their respective means: 1 on the left and 0 on the right. Its prediction vector is h₁ = (1, 0, 0).
That vector misses the target, but it supplies a useful direction.
The zero baseline keeps the geometry simple. Squared-error boosting commonly starts at the outcome mean instead.
05 / 12
Take a small step
Use a learning rate of α = ½. Add half of the tree’s prediction to the current model.
F₁ = F₀ + ½h₁
The tree supplies a direction and a proposed correction. The learning rate scales how much of that correction we use.
06 / 12
Look at what remains
Our ensemble now predicts F₁ = (½, 0, 0). The outcomes have not moved.
The dashed arrow connects our new prediction to the target. Its coordinates are the errors still to be corrected:
r₁ = y − F₁ = (½, −1, 1).
07 / 12
Make this the new origin
Place a local coordinate system at F₁. Seen from here, the target is the remaining displacement y − F₁.
We give the next tree the same input values, but new targets: these residuals. It learns how to adjust the predictions we already have.
This is a change of viewpoint. Neither the observed outcomes nor the original training inputs change.
08 / 12
Fit the next correction
The next stump chooses a different split. It groups the first two residuals together and leaves the third on its own.
Its leaf means give h₂ = (−¼, −¼, 1). We place this correction at the ensemble’s current position and again take half the proposed step.
The previous tree stays fixed. The new tree adds another function.
09 / 12
The steps add up
Two short steps give one ensemble:
F₂ = ½h₁ + ½h₂.
Adding functions adds their prediction vectors coordinate by coordinate. The path in this picture is the accumulated contribution of the trees.
At a new input, we evaluate those same fitted trees and add their contributions there too.
10 / 12
Recompute. Refit. Repeat.
After each step, recompute the residuals and fit a new stump. Move through the rounds to see the path approach the target.
A different learning rate changes the residuals left for the next tree, so it can also change which correction that tree learns.
This plot shows training loss. Predictive performance still needs validation, and later rounds can fit noise.
11 / 12
Choose the improving orientation
A candidate tree gives us a line through the current prediction. Both h and −h are valid trees: negate every leaf prediction to reverse its direction.
If h makes an obtuse angle with the residual, use −h. Every nonorthogonal line has an orientation that permits a sufficiently short improving step. At exactly 90°, neither orientation helps.
Adjust the line and step length in this flat view. The solid arrow automatically uses the improving orientation when one exists; the dashed arrow shows its opposite. Landing inside the circle reduces loss. Even with the correct orientation, a long step can overshoot.
The residual-fitted trees above already have the correct sign. Because their leaves predict mean residuals, r·h = ‖h‖² > 0 whenever h ≠ 0. This freely adjustable line isolates the geometry.
12 / 12
That direction is the negative gradient
For half-squared-error loss, the negative gradient points from our current prediction vector to the observed outcomes.
−∇L(F) = y − F.
We fit a tree to approximate that desired correction, then add a scaled copy to the current function. This is gradient descent in function space, with directions supplied by our weak learners.