Pick the right classical tool

Hook

Same nine points, same one troublemaker — a stray point labeled A that sits spatially right inside cluster B's territory. Three tools from this part, run on the exact same data. Do they agree about what to do with it?

Intuition

k=1 predicts "A" (fooled); k=3 predicts "B" (correct)

Switch lenses. kNN — short for k-nearest neighbors — and Naive Bayes both use the labels — they're trying to predict a class for a new point. DBSCAN — short for Density-Based Spatial Clustering of Applications with Noise — never looks at a single label — it just asks which points are close enough together to call a cluster, and lets the noisy point land wherever the geometry actually puts it.

Formalize

Three different questions about the same nine points and the same query at $(5, 5.5)$ :

\text{kNN: } \hat y = \text{majority vote of } k \text{ closest labeled points}

$\hat y$ — the predicted class for the query point.
$k$ — how many nearest neighbors get consulted for the vote.

\text{Naive Bayes: } \hat y = \arg\max_c P(c)\prod_i P(x_i \mid c)

$c$ — one candidate class being scored.
$P(c)$ — the prior probability of class $c$ .
$x_i$ — the $i$ -th feature (piece of evidence) of the query point.
$P(x_i \mid c)$ — the likelihood of feature $x_i$ given class $c$ .

\text{DBSCAN: cluster} = \text{whatever's density-reachable, no labels involved at all}

kNN answers with individual points
kNN answers with whichever individual points are nearest — a purely local vote among a handful of neighbors.
Naive Bayes answers with an aggregate statistic
Naive Bayes answers with an aggregate statistic across the whole class, pooling evidence from every training point that shares a label.
DBSCAN answers a different question entirely
DBSCAN doesn't answer the labeled question at all — it answers a completely different one, grouping points by density alone, with no labels involved at any point.

Play

query at (5, 5.5) — true labels shown; the noisy point is labeled A but sits inside B's territory

DBSCAN's own clusters — the noisy point joins cluster 1, same as the real B points

DBSCAN's own clustering puts the noisy point in the same cluster as the real B points — not because it disagrees with the label, but because it never consulted the label in the first place. Density doesn't care what a point is called.

Worked example

The noisy point sits at $(5.2, 5.3)$ , labeled A, right next to the real B cluster:

kNN with k=1 gets fooled
The single nearest neighbor to the query is the noisy point itself, at distance $0.28$ . Prediction: A — wrong, exactly like Chapter 1's own noisy-point example.
Naive Bayes isn't fooled at all
Binarizing position at threshold $3.5$ , the query $(5, 5.5)$ is high-x and high-y. Class A's statistics come from all five A points, not just the nearby noisy one, and (Laplace-smoothed) likelihoods for "high" on each feature are:
- $P(\text{high x}\mid A) = P(\text{high y}\mid A) = (1+1)/(5+2) \approx 0.286$ (only the noisy point is high on either feature)
- $P(\text{high x}\mid B) = P(\text{high y}\mid B) = (4+1)/(4+2) \approx 0.833$ (all four B points are high on both)
Unnormalized scores (prior times both feature likelihoods):
- A: $\frac59 \times 0.286 \times 0.286 \approx 0.045$
- B: $\frac49 \times 0.833 \times 0.833 \approx 0.309$
Normalizing: $0.309/(0.045+0.309) \approx 87\%$ for B — the noisy point's contribution gets diluted by the four genuine, clearly-not-near-B, A points.
DBSCAN doesn't use the label at all
Clustering the same nine points by density alone, the noisy point ends up in the same cluster as the four real B points — its spatial neighbors, regardless of what it was told to call itself.

Checkpoint

Pick the lens that groups the noisy point with its true spatial neighbors, entirely ignoring the label it was given.

pick a lens

Pick a lens to try it

Summary

\text{kNN: local vote} \qquad \text{Naive Bayes: aggregate statistics} \qquad \text{DBSCAN: pure density, no labels}

None of these three answers is more "correct" than the others — they're answers to three different questions, and picking the right classical tool starts with knowing which question actually needs answering. A single noisy point can fool a method that only looks locally (kNN with small $k$ ), barely register against a method that aggregates broadly (Naive Bayes), or simply not matter at all to a method that never used labels to begin with (DBSCAN). This closes Classical ML, Continued. The next part turns to a question every model built so far has dodged: not just whether a prediction is right, but how to actually measure that, honestly.