Four chapters, four ways of building a classifier from the same idea. Run them all on the exact same data — who actually wins?
This is Chapter 8's dataset, unchanged: 25 noisy training points, 24 clean validation points never used in training. Switch models and watch the fitted regions redraw completely, even though nothing about the data changed at all.
Every model here answers the same question — where's the boundary? — using tools built over the last three chapters:
- Single tree, depth 1
A single tree, pruned to depth 1 (Chapter 8).
- Same tree, depth 6
The same tree structure grown recklessly to depth 6 (Chapter 8's overfitting example).
- Bagged forest of depth-6 trees
A bagged forest of twenty depth-6 trees (Chapter 9) — the same reckless trees, averaged.
- AdaBoost with five stumps
AdaBoost — short for Adaptive Boosting — with five rounds of single-split stumps (Chapter 10).
- Same points, different rules
None of them see anything the others don't. The differences in the scoreboard come entirely from how each one turns the same 25 points into a decision rule.
| Model | Train | Validation |
|---|---|---|
| Single tree (depth 1) | 80% | 100% |
| Single tree (depth 6, overfit) | 96% | 83% |
| Bagged forest (20 depth-6 trees) | 88% | 92% |
| AdaBoost (5 rounds of stumps) | 80% | 100% |
The depth-6 tree tops the training column at 96% — and comes in dead last on validation at 83%. Bagging, built from twenty copies of that same overfit tree, can't out-train it (88%) but comfortably out-generalizes it (92%). The two techniques that actually reach 100% validation, depth-1 and boosting, do it by never overfitting in the first place — not by fixing an overfit model after the fact.
- Compare boosting's first stump to the depth-1 tree
AdaBoost's very first stump splits at — the identical threshold Chapter 8's depth-1 tree finds on its own. That's not a coincidence: both are searching for the single split that best separates the data, and there's only one true boundary here.
- See why the remaining rounds add nothing
The remaining four boosting rounds add nothing further, because the first stump already reached every validation point boosting could hope to fix.
Pick whichever model reaches the best validation accuracy on the scoreboard — 100%. More than one row can be right.
| Model | Train | Validation |
|---|---|---|
| Single tree (depth 1) | 80% | 100% |
| Single tree (depth 6, overfit) | 96% | 83% |
| Bagged forest (20 depth-6 trees) | 88% | 92% |
| AdaBoost (5 rounds of stumps) | 80% | 100% |
The scoreboard's real lesson isn't which model wins — it's why. Bagging and boosting are both insurance against picking the wrong complexity: bagging rescues an overfit model by averaging away its noise, boosting builds up from something too simple. Neither insurance policy was needed here, because the simplest reasonable model already matched the true pattern exactly. That's the bias-variance tradeoff from three chapters ago, one more time: the win was never "use a fancier technique," it was "find the right complexity" — everything else is a way of getting there when you can't just pick it by hand. This part returns later in the course, once more machinery has been built, to round out the classical toolkit with nearest-neighbor methods, density-based clustering, and a few more models this scoreboard never got to try.