Gradient Boosting

Our observations are (x₁, y₁) = (1, 1) and (x₂, y₂) = (1, 0). Each axis represents a prediction for one observation: f₁ for observation 1 and f₂ for observation 2.

The blue point is our prediction vector, initially (0, 0). The observed outcome vector y = (1, 0) stays fixed.

A residual is an observed outcome minus its prediction. Our first prediction is too low by 1; the second already matches its outcome.

r = (1 − 0, 0 − 0) = (1, 0).

The arrow shows the correction that would make both predictions exact if we could change them independently. Our predictions are still at (0, 0).

A tree uses the inputs to decide which value to predict. Both observations have exactly the same input, so they must reach the same leaf and receive the same value:

h(x₁) = h(x₂) = a,
h = (a, a).

Starting at (0, 0), any sequence of these shared updates keeps our predictions on f₁ = f₂. The horizontal residual vector (1, 0) cannot itself be predicted by a tree using this input.

Write each signed residual as rᵢ = yᵢ − fᵢ. With two observations, averaging their squared residuals gives mean squared error. Each observation contributes loss ℓ(r) = ½r²:

MSE(f) = L(f) = Σᵢℓ(rᵢ)
= ½(r₁² + r₂²).

Points at the same distance from y = (1, 0) have the same MSE. Our predictions still start at (0, 0), with MSE ½.

A residual is the observed outcome minus its prediction: r = y − f. At the starting point the residuals are 1 and 0. Since both inputs are identical, a fitted tree must predict one shared residual value. The least-squares fit is their mean.

Geometrically, this fit is the orthogonal projection of the residual vector onto the line of available updates. Write the line's direction as u = (1, 1). The projection is the vector on this line closest to r in Euclidean distance:

h = [(r · u) / (u · u)]u = (½, ½),
f_new = (0, 0) + (½, ½) = (½, ½).

As you scroll, the predictions move along the diagonal. The outcomes stay fixed while the remaining residual changes. MSE falls from ½ to ¼.

The remaining residual projects to zero on the available update line. Refitting a tree therefore gives a shared prediction of zero, so another residual update cannot move us. Geometrically, every available update has the form (a, a), and

(½, −½) · (a, a) = 0.

The equal-MSE circle touches the diagonal at this point. The faint trial points show that moving up or down the diagonal increases MSE, so (½, ½) is the best prediction this model can represent. A nonzero residual can remain even when every useful update is exhausted.

Keep the same observations and shared predictions. Change only the loss applied to each signed residual rᵢ = yᵢ − fᵢ. A positive residual means underprediction; a negative residual means overprediction.

ℓ(r) = 0.8r² if r ≥ 0;
ℓ(r) = 0.2r² if r < 0;
L(f) = Σᵢℓ(rᵢ).

At predictions (0.5, 0.5), the residuals (0.5, −0.5) give loss 0.25. At predictions (0.8, 0.8), the residuals (0.2, −0.8) give loss 0.16. Our predictions stay at (½, ½) while the circles change shape. The new contours touch the diagonal at (0.8, 0.8).

Ordinary residual fitting would leave us at (½, ½). Its zero mean no longer tells us that the selected loss is minimized. We need targets that reflect how this loss charges for each error.

Keep the predictions at f = (½, ½). With observed outcomes y = (1, 0), the ordinary residuals are y − f = (0.5, −0.5). The asymmetric squared loss gives different negative-gradient targets:

−∇L(f) = (0.8, −0.2).

A loss's gradient contains derivatives with respect to the predictions. Negating those derivatives gives targets pointing toward lower loss. Here, the target for observation i is 1.6rᵢ when rᵢ ≥ 0, and 0.4rᵢ when rᵢ < 0. For MSE in this two-observation example, the negative-gradient targets equal the ordinary residuals.

Both kinds of target require the observed outcome y. Neither can be computed from x alone for a new case whose outcome is unknown. During training, we pair each input xᵢ with its calculated target and fit a tree to learn an input-to-correction function h(x).

Once learned, that tree evaluates a new input using x alone. It predicts a correction without needing the new case's outcome. The training targets and the learned tree's predictions therefore play different roles.

The two training targets are g = −∇L(f) = (0.8, −0.2), but their inputs are identical: x₁ = x₂ = 1. Both observations must reach the same leaf. The least-squares tree fit averages the targets, projecting g onto the same available update line with direction u = (1, 1):

h = [(g · u) / (u · u)]u = (0.3, 0.3),
h(1) = (0.8 − 0.2) / 2 = 0.3.

The ordinary residual (0.5, −0.5) projects to zero here. The negative-gradient targets instead have a nonzero projection, giving an update that reduces the asymmetric loss.

The full target vector asks for different corrections at the two observations. A function of this input cannot supply those two values separately. The tree supplies the shared correction it can represent.

Use learning rate α = ½, so the actual update is half the fitted correction:

f_new = f + ½h
= (0.5, 0.5) + (0.15, 0.15)
= (0.65, 0.65).

This tree has one leaf, so its learned function is h(x) = 0.3. Evaluating it at a new input uses that input alone; the observed outcomes were needed to create its training targets.

The projection is the Euclidean least-squares fit of the targets. Loss contours describe the objective, and the learning rate scales the fitted vector afterward. For a fixed tree leaf partition, least-squares leaf means likewise project targets onto the space of vectors constant within each leaf. Searching over different tree partitions requires a search across those spaces; choosing final leaf values to minimize a different loss can also change the correction.

Start again with F₀(x) = 0.5. At every round, compute signed residuals rₘᵢ = yᵢ − Fₘ(xᵢ) from the current predictions and observed outcomes. The negative-gradient target is 1.6rₘᵢ when rₘᵢ ≥ 0, and 0.4rₘᵢ when rₘᵢ < 0. Fit a new tree using the inputs and add half its correction. Earlier trees stay fixed.

Both observations receive the same prediction Fₘ(1). Each least-squares tree fit projects the current negative-gradient vector onto the diagonal. While the shared prediction is between 0 and 1, the mean of their negative-gradient targets is 0.8 − Fₘ(1). The fitted tree and next model are therefore

hₘ₊₁(1) = 0.8 − Fₘ(1),
Fₘ₊₁(x) = Fₘ(x) + ½hₘ₊₁(x).

The remaining distance to 0.8 halves on every round. The six updates shown take the shared prediction through 0.65, 0.725, 0.7625, 0.78125, 0.790625, 0.7953125. This finite sequence approaches 0.8; it does not reach it exactly.

The resulting ensemble evaluates a new input by adding the learned functions' outputs. The outcome-dependent targets are recomputed during training, while prediction on a new case uses the fitted functions of x.

This scene inspects the exact limiting prediction (0.8, 0.8). The preceding six updates ended at (0.7953125, 0.7953125), close to this limit.

The remaining residual is (0.2, −0.8). Its mean is −0.3, so a raw-residual fit would move our predictions back toward (0.5, 0.5) and increase the selected loss. The negative-gradient targets instead are

−∇L(f) = (0.32, −0.32),
h(1) = (0.32 − 0.32) / 2 = 0.

The full negative gradient is perpendicular to every available update (a, a), so its projection onto the diagonal is zero. Its fitted shared correction is zero, and no tree using this input can improve the loss here.

The full gradient can remain nonzero at a constrained optimum. Independently changing both predictions could still reduce the loss, but our trees cannot represent the target y = (1, 0) when both inputs are identical.

The target y = (1, 0) and the attainable diagonal stay fixed. Each loss is applied to signed residuals rᵢ = yᵢ − fᵢ, then summed as L(f) = Σᵢℓ(rᵢ). The asymmetric option uses τ = 0.8, making underprediction four times as costly as overprediction. Its best attainable prediction pair is (0.8, 0.8).

Huber loss clips large gradient components. The Huber option uses δ = ¼, so every shared prediction in [¼, ¾] is optimal. Fourth-power error gives large errors more influence; symmetry keeps its optimum at (½, ½).

“Fit next update” fits the negative-gradient targets, then selects the shared correction that minimizes the chosen loss. For Huber and fourth-power error, that correction can differ from the mean gradient target.

Starting at zero exposes the update. Initialization at the selected loss's best constant would already solve this two-observation example.

Apply the same fit-and-update process to a full gradient-boosting calculation. We now use four observations and a mean starting prediction, following the example from section 1.4.3 of the boosting notes: one predictor x, with outcomes y = (0, 2, 1, 5) at x = (1, 2, 3, 4). Squared-error boosting starts at the best constant prediction, the outcome mean:

F₀(x) = (0 + 2 + 1 + 5) / 4 = 2.

The prediction vector is z₀ = (2, 2, 2, 2), and its total half-squared loss is ½(4 + 0 + 1 + 9) = 7. This loss sums over the observations; it is not a mean squared error.

The figure shows the input x and the observed response. Each blue prediction will change as we add trees; the black observed outcomes stay fixed. We use half-squared-error loss to make every gradient and update explicit.

The calculations below use α = 1. Changing the learning rate refits each subsequent tree to its new negative-gradient targets.

Write zᵢ = F(xᵢ) and regard the total loss as a function of the four prediction values:

L(z) = ½ Σᵢ₌₁⁴ (yᵢ − zᵢ)².

Hold the observed inputs and outcomes fixed. Changing the prediction zᵢ changes its contribution to the loss, giving

∂L / ∂zᵢ = zᵢ − yᵢ.

For example, the fourth derivative is ½ · 2(5 − z₄)(−1) = z₄ − 5. Each partial derivative varies one prediction while holding the other three fixed.

At the initial model, the gradient is g₁ = (2, 0, 1, −3). Its negative gives the pseudo-residual targets y − z₀ = (−2, 0, −1, 3). For observation 4, the gradient entry is 2 − 5 = −3: increasing that prediction initially reduces loss.

The derivative is with respect to a prediction, not the input x or a tree’s split threshold. For this loss, the pseudo-residuals equal ordinary residuals. Other losses give different targets. The next step is to turn these four desired corrections into a function of x.

The gradient supplies a target at each training observation. A tree supplies a correction that we can also evaluate at a new input. Here the tree is a stump: one split and two leaf values. For each candidate split, fit the mean residual on each side, then compare Σᵢ(rᵢ − h(xᵢ))². This is the residual-fitting SSE used to choose the tree.

There are three possible training partitions, represented by cuts at 1.5, 2.5, and 3.5. Their fitted SSEs are 26/3, 10, and 2. The cut at 3.5 wins.

Left mean = (−2 + 0 − 1) / 3 = −1;
right mean = 3.

The fitted correction is h₁ = (−1, −1, −1, 3). It approximates the negative gradient with two shared leaf corrections. The four negative-gradient targets ask for individual changes; a stump cannot supply four independent values. With squared error, each mean residual also minimizes the original loss within its leaf.

Inspect another candidate in the figure to see its leaf means and SSE. This comparison does not replace the fitted winner in the ensemble.

With α = 1, add the fitted correction to each current prediction:

z₁ = z₀ + h₁
= (2, 2, 2, 2) + (−1, −1, −1, 3)
= (1, 1, 1, 5).

The new loss is ½(1 + 1 + 0 + 0) = 1, down from 7. Observation 2 started with a correct prediction of 2 and now receives 1. The shared correction worsens that individual prediction while improving the aggregate fit.

The tree is also a descent direction: g₁·h₁ = −12 < 0. Along it, the exact loss is L(z₀ + αh₁) = 7 − 12α + 6α², minimized at α = 1. A useful direction still requires an appropriate step size.

Evaluate the loss gradient at the updated predictions. It is now (1, −1, 0, 0), so its negative gives fresh pseudo-residuals:

y − z₁ = (−1, 1, 0, 0).

Repeating the candidate comparison now gives SSEs 2/3, 2, and 2. The best cut has moved to 1.5. Its left leaf contains −1, while its right leaf predicts (1 + 0 + 0) / 3 = 1/3.

The second correction is therefore h₂ = (−1, 1/3, 1/3, 1/3). Tree one stays fixed; tree two learns what is still missing from the ensemble.

At α = 1/2 the residuals and second tree differ. The cuts at 1.5 and 3.5 tie; the implementation selects the first cut in increasing order.

Add tree two with α = 1:

z₂ = z₁ + h₂
= (0, 4/3, 4/3, 16/3).

The remaining residuals are (0, 2/3, −1/3, −1/3), so the total half-squared loss is ½(0 + 4/9 + 1/9 + 1/9) = 1/3. Across the two updates, loss falls 7 → 1 → 1/3.

Observation 4 was exactly right after tree one. It now moves from 5 to 16/3, because it shares tree two’s right-leaf correction. Boosting reduces a total loss; it need not improve every observation on every round.

Each gradient-boosting round follows the same sequence: compute a negative gradient, approximate it with a fitted tree, add a scaled correction, and recompute. These calculations establish training-loss reductions; performance on new observations still requires validation.

Continue from the same constant model. At round m, evaluate the current ensemble on the training data, compute a fresh negative gradient, fit the next tree, and add its scaled correction.

rₘᵢ = yᵢ − Fₘ₋₁(xᵢ),
Fₘ(x) = Fₘ₋₁(x) + αhₘ(x).

Move through the rounds to see the cumulative prediction. The earlier trees stay fixed; each new tree fits what the current ensemble still needs. Changing α changes the residual targets and can change later splits.

At a new input, prediction uses the same fitted functions: Fₘ(x) = 2 + αΣₖ₌₁ᵐ hₖ(x). We add their outputs, rather than choosing one tree or taking a vote.

This figure reports training loss. The learning rate, tree complexity, and number of rounds govern how the model grows. Performance on new observations needs validation; more rounds can fit noise.

For a differentiable loss ℓ(y, z), gradient boosting starts from a constant that minimizes total loss, when a finite minimizer exists. Otherwise, constrain the starting constant to a finite closed interval or use a suitable penalty. Each round then computes

rₘᵢ = −∂ℓ(yᵢ, z)/∂z
evaluated at z = Fₘ₋₁(xᵢ).

These pseudo-residuals describe the desired change at each training prediction. Fit a regression tree to them to choose its regions. Within each region, choose a leaf correction that minimizes the original loss when added to the current model. Use a learning rate 0 < α ≤ 1 to scale the correction tree, add it, and differentiate again.

For half-squared error, both calculations simplify: pseudo-residuals are ordinary residuals, and the optimal leaf correction is their mean. For other losses, the gradient targets and final leaf values can differ.

The tree is a constrained approximation to a gradient direction. Gradient boosting therefore performs stagewise optimization over functions that the chosen trees can represent.

Change the loss and compare the actual fits.

Keep the regression problem and change the loss. The new example has x = (1, …, 7) and y = (0, 0, 1, 2, 3, 4, 15). The unusually high seventh outcome lets us see how the loss controls its influence. You can also return to the original four observations.

Write the ordinary residual as e = y − z. Squared error gives the negative-gradient target e. Huber loss is quadratic near zero and linear in the tails, giving

−∂ℓ_δ/∂z = clip(e, −δ, δ).

At the Huber-optimal constant F₀ = 2, the last residual is 13 but its target is 1 when δ = 1. Its influence on the split search is bounded. With asymmetric squared loss, underprediction and overprediction instead receive different weights:

−∂ℓ_τ/∂z = 2τe if e ≥ 0;
2(1 − τ)e if e < 0.

At τ = 0.8, an equally large underprediction receives four times the gradient magnitude of an overprediction. This loss estimates an expectile, rather than a quantile. At τ = 0.5, it becomes our half-squared loss.

Changing the loss refits both paths from the same loss-optimal constant. At later rounds this target comparison uses the current gradient-boosted model. The earlier Euclidean circles describe squared error only.

The next stump still uses least squares to approximate its training targets. Compare the split search using ordinary residuals with the split search using the selected loss’s negative gradients. Each column reports its own target-fitting SSE; their numerical scales need not agree.

At the initial seven-point Huber model with δ = 1, raw residuals select a cut at 6.5, isolating the outcome of 15. The negative-gradient targets (−1, −1, −1, 0, 1, 1, 1) select 3.5. Clipping changes which observations drive the partition, even though the data and current predictions are identical.

This is where the gradient calculation matters for tree structure. Averaging raw residuals and calling the result “Huber boosting” would continue to select splits for the squared-error objective.

Cuts 3.5 and 4.5 tie in the initial Huber gradient fit; the first split wins ties in ascending order. At every round, both split searches use the same current gradient-boosted prediction. This isolates the effect of changing the targets. A gradient-selected split need not produce the largest possible one-round reduction in the original loss.

The fitted gradient tree determines two regions. Its mean target in each region describes the gradient approximation, but the prediction correction has a separate job. Choose the correction γ that minimizes the selected loss within that region:

γⱼ = arg min_γ Σ_{i ∈ Rⱼ} ℓ(yᵢ, F(xᵢ) + γ).

For the initial Huber split at 3.5, the target means are −1 and 3/4. The loss-minimizing prediction corrections are instead −5/3 and 3/2. The curves show the original leaf losses and mark both values. The learning rate scales the minimizing corrections afterward.

Squared error makes both values the mean residual. Huber corrections solve Σ clip(eᵢ − γ, −δ, δ) = 0; asymmetric squared corrections solve the corresponding weighted residual equation. Changing the loss can therefore change both the tree’s regions and the sizes of its updates.

The curves use the next gradient-selected stump at the chosen round. Huber can have an interval of minimizing corrections; the calculation chooses its midpoint.

Both methods start at the same constant and use the same learning rate and stump size. Gradient boosting fits the negative gradients and recalibrates its leaves with the chosen loss. The residual shortcut fits y − F and adds the mean residual in each leaf.

Move through the rounds and switch losses. Under squared error, the paths coincide. Other losses can produce different splits, corrections, predictions, and training-loss histories. The graph evaluates both paths with the currently selected loss; loss values across different loss choices are not directly comparable.

These are finite sequences of greedy fits, so the gradient path need not have lower loss at every round. A large residual can still eventually be fitted under Huber loss, especially when a tree can isolate it. Clipping its gradient does not exclude that observation from the model.

Select “Identical inputs” for a sharper distinction. The three responses are (0, 0, 10), all at x = 1, so every tree must give them one shared prediction. With Huber δ = 1, the best constant is 1/2. Gradient boosting stays there. The residual shortcut moves toward the response mean, 10/3, and increases Huber loss. A zero mean residual does not imply a zero gradient for this loss.

Forward stagewise optimization can itself minimize a general loss. The comparison here isolates the ordinary least-squares residual-fitting shortcut. The useful generalization is to derive the target and leaf update from the objective we actually want to optimize.

All displayed losses sum over the observations. Half-squared error is proportional to MSE, with identical minimizers. These are training fits; they do not establish performance on new observations.