k-fold & stratified cross-validation

Ten points trend almost perfectly along a line — except one, which sits far off it. Step through five different train/test splits of the same ten points and watch the held-out error swing from barely-there to enormous, depending only on whether that one point happened to land in the test set.

k-fold cross-validation partitions the data into $k$ equal folds. For each fold in turn, train on the other $k-1$ folds and evaluate on the one held out:

• CV error — the k-fold cross-validation estimate of error: the average held-out error across every fold.
• $k$ — the number of folds the data is split into.
• $i$ — the index of the current fold, ranging from $1$ to $k$.
• MSE — mean squared error: the metric used to score each fold's held-out predictions.

1. A single split is one sample from the process

   A single train/test split gives you one sample from this process — one particular way the anomaly (or any unusual point) could land, for better or worse.

2. Repeating it reveals disagreement, not just a mean

   Cross-validation runs the experiment $k$ times and reports both the mean and how much the folds disagreed — a spread that a single split can never reveal about itself.

Four of the five folds score a modest error. The one fold holding out the anomaly alone scores dramatically worse — over 30 times higher than the best fold. The mean folds all five outcomes into a single honest number, and the stdev quantifies exactly how much that number should be trusted.

Ten points following $y = 2x + 1$, except $x=6$, where $y=30$ instead of the line's $13$:

1. Most folds barely notice the anomaly

   Holding out $x=3,4$, fitting on the other 8 (including the anomaly) recovers $y\approx2x+3.125$ — the anomaly pulls the intercept up slightly from the clean line's $1$, but the slope stays exactly $2$. Predicting the held-out points:

   • $x=3$: predicted $\approx9.125$, actual $7$, squared error $\approx4.52$
   • $x=4$: predicted $\approx11.125$, actual $9$, squared error $\approx4.52$

   MSE $=(4.52+4.52)/2\approx4.52$ — the held-out points still land close to the fitted line.

2. One fold holds out the anomaly itself

   Holding out $x=5,6$ trains on 8 clean points, recovering the line almost exactly ($y = 2x+1$). Predicting the held-out points:

   • $x=5$: predicted $11$, actual $11$, squared error $0$
   • $x=6$: predicted $13$, actual $30$ (the anomaly), squared error $17^2=289$

   MSE $=(0+289)/2=144.5$.

3. One number, or five, or the truth

   Running the same fit-and-hold-out recipe for the remaining three folds gives each fold's MSE:

   • Fold (holdout $x=1,2$): $\approx9.85$
   • Fold (holdout $x=3,4$): $\approx4.52$ (shown above)
   • Fold (holdout $x=5,6$): $144.5$ (shown above)
   • Fold (holdout $x=7,8$): $\approx7.39$
   • Fold (holdout $x=9,10$): $\approx26.73$

   Mean $=(9.85+4.52+144.5+7.39+26.73)/5\approx38.60$; stdev across those five values $\approx59.83$ — a spread almost as large as the mean itself. A single 80/20 split (train on the first 8, test on the last 2) is exactly the holdout-$x{=}9,10$ fold above, scoring $26.73$: not wrong, but silent about how much luckier or unluckier it could have been.