Grade and audit a machine learning pipeline honestly

One scored classifier, ten examples. Slide the decision threshold across three candidates and watch accuracy, precision, recall, and F1 — short for the F1 score — move independently — sometimes together, sometimes in opposite directions.

A real evaluation doesn't pick one metric — it runs the whole scoreboard and asks where the numbers agree and where they don't:

1. Each metric was built in isolation

   Every one of these — accuracy, F1, AUC, per-fold accuracy, best-threshold F1 — was built in its own chapter, looking at the classifier through only one lens at a time.

2. None of them is the right answer

   Each answers a different question, and a model that looks good on one can look mediocre on another — the point of a real evaluation is to run all of them and compare, not to pick a favorite.

At the default threshold of $0.5$, accuracy is a respectable $0.7$ but F1 lags at $0.73$. The ranking AUC — short for Area Under the Curve — of $0.76$ says the model's raw ordering of examples is decent but not sharp. And the fold breakdown shows exactly why: two folds are trivially easy (accuracy $1.0$), while three folds straddling the decision boundary score only $0.5$ each — the same overall accuracy, but now you know precisely where the model is guessing.

Ten scored examples, five positive and five negative, with three overlapping scores right around the boundary:

1. One threshold, two disagreeing scores

   At threshold $0.5$, all six examples scoring $\geq0.5$ get predicted positive:

   • TP (true positives) $=4$
   • FP (false positives) $=2$
   • FN (false negatives) $=1$
   • TN (true negatives) $=3$

   Accuracy $=0.7$ looks fine. But precision is only $2/3$ and F1 lands at $8/11 \approx 0.727$ — the two false alarms cost more in F1 than they cost in accuracy.

2. A threshold-free verdict

   Ranking every pair of one positive against one negative, the positive scores higher in 19 of 25 pairs: AUC $=19/25=0.76$. This doesn't move if the deployed threshold changes — it's a property of the scores themselves, not of wherever the cutoff happens to sit.

3. Where the fixed threshold actually fails

   Splitting the ten examples into five folds of two, in score order, and scoring the same threshold-$0.5$ classifier on each:

   • Fold 1 (scores $0.9, 0.8$, both label $1$): both predicted $1$ — accuracy $1.0$
   • Fold 2 (scores $0.7$ label $0$, $0.65$ label $1$): both predicted $1$, so the label-$0$ one is wrong — accuracy $0.5$
   • Fold 3 (scores $0.6$ label $0$, $0.55$ label $1$): both predicted $1$, same pattern — accuracy $0.5$
   • Fold 4 (scores $0.4$ label $0$, $0.3$ label $1$): both predicted $0$, so the label-$1$ one is wrong — accuracy $0.5$
   • Fold 5 (scores $0.2, 0.1$, both label $0$): both predicted $0$ — accuracy $1.0$

   $[1.0, 0.5, 0.5, 0.5, 1.0]$ — mean $0.7$, matching the aggregate exactly, but now visibly concentrated in three folds where a positive and a negative sit within $0.1$ of each other.

4. A better threshold, found by search

   Grid searching every candidate threshold by F1 finds $0.3$ scores $10/13 \approx 0.769$ — better than the default's $0.727$, at the cost of accepting three false positives instead of two. Whether that trade is worth it depends on what the model is for, which no metric in this part can answer by itself.