The naive Bayes classifier

Six training messages, three spam, three not. Toggle smoothing on a new message containing "free" and "money" — both toggles predict spam, but look at how confident each one is. One of them is lying about how sure it should be.

Naive Bayes multiplies Bayes' rule across every feature, assuming they're independent given the class — the "naive" part:

• class — the category being predicted, such as spam or not-spam.
• $x_1,\dots,x_n$ — the $n$ individual pieces of evidence, such as which words are present.
• $P(\text{class})$ — the prior probability of the class, before looking at any evidence.
• $P(x_i \mid \text{class})$ — the likelihood of feature $i$: how often it shows up in training examples of that class.
• $n$ — the number of features being combined.

1. Each likelihood is just a frequency

   $P(x_i \mid \text{class})$ is how often word $i$ showed up in training messages of that class — a simple count, nothing more elaborate.

2. Independence turns a hard problem into easy ones

   Multiplying independent probabilities together is exactly what "assume independence" means — it's the one assumption that turns a hard joint-probability problem into a handful of easy word-counting ones.

Build a message by toggling which words are present. "Meeting" alone should pull hard toward not-spam; "free" alone pulls the other way. Watch how the posterior shifts as you add or remove one piece of evidence at a time — each word's presence (or absence) multiplies in independently.

"Free" never appears in any of the three not-spam training messages:

1. The zero-frequency problem

   $P(\text{free}\mid\text{not spam}) = 0/3 = 0$ exactly. For a message containing "free," this makes the entire not-spam probability collapse to precisely $0$ — not small, not unlikely, but mathematically impossible — purely because of one gap in a tiny training set.

2. Laplace smoothing patches it

   Add one fake "seen" and one fake "not seen" example to every count: $P(\text{free}\mid\text{not spam}) = (0+1)/(3+2) = 0.2$. Never exactly zero again, no matter how sparse the training data.

3. Same prediction, honest confidence

   For "free, money" (meeting absent), each class's unnormalized score is its prior times $P(\text{free}\mid c) \times P(\text{money}\mid c) \times (1-P(\text{meeting}\mid c))$:

   Unsmoothed:

   • Spam: $0.5 \times \frac23 \times \frac23 \times (1-0) \approx 0.222$
   • Not spam: $0.5 \times 0 \times \frac13 \times (1-\frac23) = 0$ — the zero-frequency word forces the whole product to $0$

   Normalizing: spam $\approx0.222/0.222=100\%$, not spam $=0\%$ — absolute certainty from one missing example.

   Smoothed:

   • Spam: $0.5 \times 0.6 \times 0.6 \times (1-0.2) = 0.144$
   • Not spam: $0.5 \times 0.2 \times 0.4 \times (1-0.6) = 0.016$

   Normalizing: spam $=0.144/0.16=90\%$, not spam $=0.016/0.16=10\%$ — the same correct prediction, but at a far more reasonable confidence.