Every classifier in this course has answered "what is this?" A detector has to answer "what, and exactly where?" — and "where" needs its own metric entirely, one that a confusion matrix alone was never built to measure.
A ground-truth box and a predicted box on a small pixel grid. Shift the prediction and watch how much they overlap — and how sharply that overlap score drops once the boxes stop lining up.
Intersection over Union (IoU) scores how well two boxes overlap, independent of their size or position:
- A detection only counts past a threshold
A detection typically only "counts" if IoU clears a threshold — commonly .
- Segmentation asks the same question per pixel
Segmentation asks the same overlap question at the level of individual pixels instead of a single box, scored with the same TP (True Positive)/FP (False Positive)/FN (False Negative) machinery as any confusion matrix: every pixel is either correctly inside the mask, correctly outside it, or wrong one way or the other.
The exact same two boxes, scored two ways: IoU treats the whole box as one geometric shape, while precision, recall, and Dice treat every pixel as its own tiny classification. Both agree the boxes don't fully align — the pixel view just says which pixels are the problem.
Ground truth: a box. Predicted box, same size, shifted right by 0, 1, or 2 columns:
- Perfect alignment: IoU=1
Shift : the boxes coincide exactly. Intersection , union , .
- Right at the standard threshold
Shift : intersection drops to pixels, union rises to . — sitting exactly on the boundary most detection benchmarks use to call something a match at all.
- Pixel metrics agree, and go further
At the same shift, the overlap (intersection) is pixels, out of in each box:
- TP (in both boxes) intersection
- FP (in predicted only)
- FN (in ground truth only)
So precision recall , and Dice — consistent with the box IoU of via , but reporting exactly which pixels were false alarms versus which were missed.
Find the shift that puts the detection right at the edge of counting as a match (IoU ≥ 0.5) — not a perfect one.
Detection and segmentation both reduce to the same question — how much do two regions overlap — just measured at different granularities: one box versus the other, or one pixel versus the other. Every evaluation idea from Part IX still applies here; IoU and Dice are just a confusion matrix wearing a geometric hat. The capstone that closes this part puts every architecture built so far — attention, residual connections, transfer learning — to work fine-tuning one real pretrained model.