Modern boosters: XGBoost & LightGBM

The faint curve is last chapter's plain GBM fit. Drag $\lambda$ up and watch the bold curve pull back toward round 1's fit — regularization is literally the ensemble refusing to fully trust round 2's correction.

XGBoost and LightGBM both replace "leaf value = mean of residuals" with a second-order (Newton) step on the loss, using each point's gradient and its curvature (Hessian):

• $w^*_{\text{leaf}}$ — the leaf's optimal prediction, replacing "mean of residuals."
• $g_i$ — point $i$'s gradient of the loss with respect to the current prediction; for squared error, $g_i = F(x_i) - y_i$.
• $h_i$ — point $i$'s Hessian (second derivative) of the loss — how sharply the loss curves at the current prediction.
• $\lambda$ — an L2 regularization strength penalizing large leaf values directly.

1. Squared error's Hessian is a constant, which is why GBM could skip it

   For squared-error loss, $h_i = 1$ for every point, always. At $\lambda = 0$ that makes $w^* = -G/H = -\big(\sum g_i\big)/n =$ mean of residuals — exactly plain gradient boosting's leaf value. Newton boosting doesn't replace GBM here; it generalizes it.

2. Regularization shrinks every leaf toward zero, not just the extreme ones

   Increasing $\lambda$ grows the denominator for every leaf, pulling $w^*$ toward 0 regardless of how large $G$ is — a direct brake on how far any single leaf is allowed to move the prediction.

3. Histogram binning speeds up finding the split, without changing what's optimized

   LightGBM (and XGBoost's histogram mode) bucket each feature into a fixed number of bins before searching for the best split, trading a small amount of split precision for a search that's proportional to the number of bins instead of the number of distinct values — the leaf-value formula above is unaffected.

Both $\lambda$ and a toy per-point Hessian are live. Raising either one shrinks the leaf values — a real curvature signal and an artificial regularization penalty end up looking identical in the formula, which is exactly the point: $H$ and $\lambda$ enter the denominator the same way.

Reusing last chapter's round 2 exactly: splitting at $x = 4.5$ gives a left leaf of 4 points with gradients $g = [0.5, -0.5, 2, 1]$ and a right leaf of 1 point with $g = -3$.

1. Left leaf: G = 3, H = 4 (at hessian=1)

   $G = 0.5 - 0.5 + 2 + 1 = 3$. With $\lambda = 0$: $w^* = -3/4 = -0.75$ — identical to plain GBM's leaf value.

2. Add regularization: lambda = 1

   $w^* = -3/(4+1) = -0.6$ — a smaller step than before, exactly as $\lambda$ is supposed to do.

3. Add curvature: toy Hessian = 2 per point

   Now $H = 4 \times 2 = 8$.

   • At $\lambda=0$: $w^* = -3/8 = -0.375$.
   • At $\lambda=1$: $w^* = -3/9 = -1/3$.
4. The right leaf shrinks the same way

   $G = -3$, and $H = 1 \times h$ for whatever the per-point Hessian $h$ is:

   • $h=1,\lambda=0$: $w^*=-(-3)/(1+0)=3$ (matches GBM)
   • $h=1,\lambda=1$: $w^*=3/(1+1)=1.5$
   • $h=2,\lambda=1$: $H=1\times2=2$, so $w^*=3/(2+1)=1$