Bias-variance decomposition

Hook

If you retrained the exact same model on a slightly different sample of the same data, would it come back with the same answer?

Intuition

Degree 0 fit to one noisy sample (dashed = the true curve)

degree = 0

Drag degree up. At 0 or 1, the fit is too stiff to follow the true curve's bend — it's wrong in the same way no matter which noisy sample you'd trained on. By degree 4, it bends through every single point exactly, chasing this sample's specific noise instead of the shape underneath it.

Formalize

Retrain that same degree on several different noisy samples of the same underlying data, and two separate error sources fall out:

\text{bias}^2 = \big(\mathbb{E}[\hat f(x)] - f(x)\big)^2, \qquad \text{variance} = \mathbb{E}\big[(\hat f(x) - \mathbb{E}[\hat f(x)])^2\big]

$x$ — a fixed input point, evaluated across many resampled fits.
$f(x)$ — the true underlying function's value at $x$ .
$\hat f(x)$ — one trained model's prediction at $x$ .
$\mathbb{E}[\hat f(x)]$ — the average prediction at $x$ , across many resampled fits.

Bias: how wrong the average fit is
Bias is how wrong the average fit is — high when the model is too simple to represent the true shape.
Variance: how much the fit swings
Variance is how much the fit swings from one sample to the next — high when the model is flexible enough to chase noise.
Total error decomposes into both
Total expected error is bias² + variance (plus noise no model can remove).

Play

Degree 0 — bias² = 3.300, variance = 0.003

degree = 0

Now all six resampled fits are drawn at once. At low degree they nearly overlap — consistently wrong, low variance — but sit visibly off the true dashed curve: that gap is bias. At high degree they fan out wildly from each other, each one perfectly tracing its own sample's noise: high variance, but their average lands close to the truth.

Worked example

The true curve here is exactly quadratic.

Degree 0 — too stiff
Just the mean, across the six samples:
- bias² $= 3.3$ — it structurally cannot bend, so it's consistently wrong by roughly the same amount every time
- variance is a tiny $0.003$
Degree 4 — too flexible
With exactly enough parameters to hit all five points exactly:
- bias² drops to $0$
- variance explodes to $0.173$ , nearly 50 times higher
Degree 2 — matches the true shape
- Bias² $\approx 0$
- a modest variance of just $0.046$
The lowest total error of any degree tried.

Checkpoint

Find the degree that minimizes total error (bias² + variance) — neither the stiffest nor the most flexible model wins.

Degree 0 — bias² = 3.300, variance = 0.003, total = 3.303

degree = 0

Move the degree slider to try it

Summary

\text{bias}^2 = \big(\mathbb{E}[\hat f(x)] - f(x)\big)^2, \qquad \text{variance} = \mathbb{E}\big[(\hat f(x) - \mathbb{E}[\hat f(x)])^2\big]

Every model choice in this book so far — tree depth, forest size, boosting rounds, polynomial degree — has secretly been tuning this same tradeoff. Too simple underfits with high bias; too flexible overfits with high variance. The next chapter looks at a direct way to fight the variance side, without giving up flexibility outright.