Correct residual errors with boosting.

A correction can be more useful than a fresh guess

Suppose a model predicts a petal length of 4 cm, but the measured length is 3 cm. The residual, observed minus predicted, is −1 cm. A negative residual asks for a lower prediction; a positive residual asks for a higher one.

A boosted model starts with a simple prediction and adds learned corrections. Unlike the forest lesson, its trees do not vote on a category. They are fitted in sequence to numerical residuals, and their outputs are added. The next tree depends on what the earlier trees have already learned.

Start the lab at stage zero. It predicts the same training-mean petal length for every sepal length. Add one tree and inspect a validation row. Read the preceding prediction, the tree’s correction, and the new prediction in the same trace.

Fit a small tree to what remains

Here is a separate worked example with six training flowers. Their sepal lengths are 4, 4.5, 5, 6, 6.5 and 7 cm; their petal lengths are 1, 1, 1, 5, 5 and 5 cm. The mean petal length is 3 cm. The first three residuals are −2 cm and the last three are +2 cm.

A depth-one tree asks whether sepal length is at most 5.5 cm. Its left leaf predicts residual −2 cm; its right leaf predicts +2 cm. Those outputs are corrections, not complete predicted flower lengths. A correction can be negative even though an actual petal length cannot.

For this squared-error lesson, each leaf outputs the average residual of the training rows that reach it. Candidate splits are compared by the sum of squared distances from those leaf means. The lab requires at least three training rows per child and tries midpoints between distinct observed sepal lengths. It keeps the first equally good threshold. If a split cannot improve squared error, the node stays a leaf.

Shrink the correction before adding it

A large correction can fit training details too quickly. Shrinkage, also called a learning rate, multiplies each new tree’s output by a number between 0 and 1. With shrinkage 0.2, the worked example’s left prediction becomes 3 + 0.2 × (−2) = 2.6 cm. The right prediction becomes 3 + 0.2 × 2 = 3.4 cm.

$F_m(x)=F_{m-1}(x)+\eta h_m(x)$

x is the sepal length in cm. m is the stage number, starting at 1 for the first tree. Fₘ₋₁(x) is the preceding prediction in cm; hₘ(x) is the new residual tree’s output in cm; η (eta) is the unitless shrinkage multiplier. Fₘ(x) is the updated petal-length prediction in cm. Read the formula as “keep the previous prediction and add a scaled correction.” Stage zero, F₀, is the mean of the training targets.

The next tree must use the new residuals. In the example, after the 0.2 step the residuals are −1.6 cm and +1.6 cm. Fitting again to the original −2/+2 residuals would not be the same learning procedure. The lab recomputes residuals from the complete preceding model before every stage.

Optional: why this is gradient boosting for squared error

For one training row, define half-squared loss as ℓ = ½(y − F)², where y is the observed target and F is the current prediction, both in cm. The derivative with respect to the prediction is F − y; its negative is y − F, the residual. A small tree approximates those negative-gradient values across training inputs. Adding its scaled predictions therefore moves in a direction chosen to reduce this loss. The factor ½ simplifies the derivative; the displayed MSE omits it. This lesson implements squared-error regression boosting. Classification boosting requires its own loss and update rule; these numerical tree outputs are not probabilities.

Explain the staircase and the error curves

Each tree gives the same correction to all inputs in one leaf. Adding trees can introduce more split locations, so the final predictor is a staircase. Inspect its exact intervals on the fixed-scale prediction plot. Vertical joins mark discontinuities; they are not smooth interpolation between values. A value equal to a threshold follows the left branch.

The second plot compares training and validation mean squared error, measured in cm². For two residuals −1 cm and +3 cm, MSE = (1² + 3²)/2 = 5 cm². Larger mistakes contribute disproportionately. Training residuals fit the next tree; validation labels only measure the result. A stage that reduces average training loss can worsen validation error or a particular validation example.

With the six-flower example and shrinkage 0.2, the initial MSE is 4 cm². After one tree, all residual magnitudes are 1.6 cm, so MSE is 2.56 cm². The real-data lab need not show these exact values: its flowers, targets and fitted splits are different.

Choose how much complexity is useful

Shrinkage, tree depth and stage count work together. Smaller shrinkage usually requires more stages to make a comparable change. Deeper trees can fit finer interactions or intervals within a stage, but can also chase idiosyncrasies. More stages are not guaranteed to improve new examples.

The Inspect lowest validation MSE so far button selects the best observed prefix among built stages, including the mean baseline. Ties retain the earlier stage. Inspecting a prefix changes the displayed model without changing the fitted sequence. Adding more stages continues from the entire built sequence, not from the prefix you are inspecting.

Before experimenting, declare at most three rate/depth configurations to compare and allow each up to 40 stages. Record every comparison, not just the winner. Forty depth-three trees are not equal compute to forty depth-one trees. Use the measured timings as observations from your current device, not universal performance claims. Choosing repeatedly on these validation rows consumes development evidence; an untouched final evaluation is still needed before a stronger claim.

Compare against something simpler on the same data

The lab predicts petal length from sepal length using the original licensed Iris data. Identical four-measurement groups stay together across the split, and one representative per group leaves 98 training and 51 validation rows. Groups are not verified specimen identities; removing duplicate vectors does not establish full statistical independence. Measurements are unaltered, and the species mixture can explain part of their association.

Both the constant mean and the straight line are fitted on those same training rows. Compare their validation MSE with the boosted prefix before claiming that extra complexity helps. Do not compare these regression MSEs directly with the synthetic forest lesson’s classification error percentages: they use different targets, datasets and units.

The local Iris manifest records Fisher (1936), the UCI source, CC BY 4.0 license, byte size and hash. The linear-regression lesson uses the same split and units.

Make the experiment useful beyond the widget

In your comparison note, identify the selected prefix, shrinkage, depth, baseline errors, and a validation case where a correction helped or hurt. State the number of configurations you tried and the evidence still needed.

For implementation practice, independently fit a residual stump, test its threshold equality rule, and verify that adding later trees does not change earlier prefix predictions. For decision practice, justify whether any observed validation improvement warrants the extra complexity, and specify what new evaluation would test that claim. The exported note is a practice artifact. It does not establish independent implementation competence, replace the planned cumulative classical-ML project, or complete human review and learner pilots.