Build a high-performance tabular ML pipeline

This is the same 25-point noisy training set and 24-point clean validation set from the trees chapters, unchanged. Switch models and watch the fitted regions redraw completely, even though nothing about the data did.

Every model here answers the same question — where's the boundary at $x=9.5$? — but each one combines evidence its own way:

• $\hat{y}(x)$ — the model's final predicted label at input $x$.
• $h_m(x)$ — round $m$'s stump prediction, for the boosted ensemble.
• $\gamma$ — the kernel's bandwidth: how quickly similarity falls off with distance.

1. Random Forest

   Twenty bagged, individually-overfit depth-6 trees, majority-voted (Chapter 3).

2. XGBoost-style booster

   A squared-loss, Newton-boosted stump ensemble — five rounds, split by the exact gain formula $G_L^2/(H_L+\lambda) + G_R^2/(H_R+\lambda)$ from Chapter 6 — run once with no regularization and once with $\lambda=1$.

3. Kernel SVM (simplified)

   Every training point votes with an RBF-kernel-weighted sign, run once with a too-narrow bandwidth ($\gamma=1$) and once tuned ($\gamma=0.1$). A real SVM solves for a sparse set of support-vector weights; this keeps the essential mechanism — similarity-weighted, non-linear voting — at toy scale.

The too-narrow kernel SVM ($\gamma=1$) tops the training column at 100% — and comes in dead last on validation at 79%. It memorized the training set's five mislabeled points instead of the boundary between them. Random Forest can't out-train it (88%) but comfortably out-generalizes it (92%). Only the two regularized configurations — XGBoost with $\lambda=1$, and the kernel tuned to $\gamma=0.1$ — reach a perfect 100% validation, and both do it by explicitly damping how far a single point is allowed to move the decision.

1. XGBoost round 1 finds the exact boundary, with clean numbers

   Splitting at $x=9.5$ (every point's toy Hessian is 1):

   • Left leaf: 10 points, $G_L=6$, $H_L=10$, so at $\lambda=0$: $w_L^* = -6/10 = -0.6$
   • Right leaf: 15 points, $G_R=-9$, $H_R=15$, so $w_R^* = 9/15 = 0.6$
   • Gain: $6^2/10 + 9^2/15 = 3.6 + 5.4 = 9$, beating every other candidate threshold outright
2. Regularization pulls both leaf values toward zero

   With $\lambda=1$ instead: $w_L^* = -6/11 \approx -0.545$ and $w_R^* = 9/16 = 0.5625$ — both leaves take a smaller step. Small enough, across five rounds, that the ensemble never chases the noise into a wrong answer, landing at 100% validation instead of the unregularized version's 96%.

3. A too-narrow kernel gets fooled by exactly one point

   At the clean validation point $x=15.25$ (true label B), $\gamma=1$'s kernel score comes out to $\approx -0.106$ — negative, predicting A — because the single nearby mislabeled training point at $x=15$ dominates a kernel that's too narrow to look past its immediate neighbor.

4. Widening the kernel fixes it, the same way weighting by shard size fixed averaging

   At $\gamma=0.1$, the same point's score is $\approx 3.05$ — solidly positive, predicting B correctly — because a wider kernel pools evidence from the many genuine B points surrounding that one noisy A, instead of trusting its nearest neighbor alone.