Part VII — Model Evaluation, Validation & Feature Engineering · Chapter 1

Confusion matrix, precision, recall & F1

Hook

Every classifier in this course has reported "accuracy" as if it settles the question of whether a model is any good. Two classifiers can score the exact same accuracy while doing completely different — and completely differently useful — things.

Intuition

Ten cases, nine negative and one positive — a rare event, like a rare disease or a fraud attempt. Toggle between a classifier that always says "negative" and one that actually catches the rare case. Watch the readout: their true/false positive/negative counts are entirely different.

Formalize

A confusion matrix counts four outcomes: true/false positive, true/false negative. From those four numbers:

precision=TPTP+FP,recall=TPTP+FN,F1=2⋅precision⋅recallprecision+recall\text{precision} = \frac{TP}{TP+FP}, \qquad \text{recall} = \frac{TP}{TP+FN}, \qquad F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision}+\text{recall}}
  • TPTP — true positives: cases correctly flagged positive.
  • FPFP — false positives: negative cases wrongly flagged positive.
  • FNFN — false negatives: positive cases wrongly flagged negative.
  • precision — of everything flagged positive, the fraction that's actually positive.
  • recall — of everything actually positive, the fraction the model caught.
  • F1F_1 — the harmonic mean of precision and recall, only high when both are high.
  1. Precision and recall ask different questions

    Precision asks "of everything I flagged positive, how much was actually positive?" Recall asks "of everything actually positive, how much did I catch?" A model can score high on one while scoring low on the other.

  2. F1 needs both to be high at once

    The F1F_1, short for the F1 score, is the harmonic mean of precision and recall — a single number that only looks good when both are reasonably high, unlike an average that a lopsided pair could still inflate.

Play

Two classifiers, read side by side. One never predicts positive at all; the other catches the single true positive at the cost of one false alarm. Precision, recall, and F1 tell them apart completely — accuracy alone can't.

Worked example

Nine negatives, one positive. Classifier A always predicts negative; classifier B predicts positive on the true case plus one extra:

  1. Both classifiers score exactly 90% accuracy

    A gets 99 true negatives and 11 false negative: 9/10=90%9/10 = 90\%. B gets 11 true positive, 11 false positive, 88 true negatives: (1+8)/10=90%(1+8)/10 = 90\% too — identical, despite behaving nothing alike.

  2. Precision and recall tell a completely different story
    • Classifier A: precision =0=0, recall =0=0 — it never predicted positive, so both are undefined-in-spirit and exactly zero by convention
    • Classifier B: precision =1/2=0.5=1/2=0.5, recall =1/1=1.0=1/1=1.0 — it caught the one case that mattered, at the cost of one false alarm
  3. F1 makes the difference impossible to miss

    A's F1 is exactly 00. B's F1 is 2(0.5)(1)/(0.5+1)=2/3≈0.6672(0.5)(1)/(0.5+1) = 2/3 \approx 0.667 — a large, clear gap that "90% accuracy" alone completely hid.

Checkpoint

Find the classifier whose F1 score clears 0.5 — accuracy alone won't tell you which one.

Pick a classifier to try it
Summary
precision=TPTP+FP,recall=TPTP+FN,F1=2⋅precision⋅recallprecision+recall\text{precision} = \frac{TP}{TP+FP}, \qquad \text{recall} = \frac{TP}{TP+FN}, \qquad F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision}+\text{recall}}

Accuracy treats every correct prediction as equally valuable — which is exactly wrong whenever one class is rare and getting it right matters more than getting the common class right. This is precisely the kind of imbalance that shows up in fraud detection, rare-disease screening, and search-and-rescue triage, and it's exactly the pattern Part V's interpretability-accuracy-tradeoff chapter gestured at without naming: the score itself can be a choice, not a neutral fact about the model.