Encoding categorical variables

Same six rows, same city column, three different ways to turn it into numbers. Ordinal encoding just assigns each city an integer. One-hot spreads it into a binary column per city. Naive target encoding replaces each city with the average outcome for that city — watch the lone SF row especially closely as you switch to it.

Three ways to turn a category into numbers, each with a different failure mode:

• $c$ — a category value, e.g. "SF".
• index of $c$ — ordinal encoding's arbitrary position for that category in some fixed list.
• $y_i$ — the label of row $i$.
• target$(c)$ — the mean label among rows sharing category $c$; naive target encoding includes the current row in that average, leave-one-out excludes it.

1. Ordinal encoding invents an ordering

   Mapping LA→0, NYC→1, SF→2 makes SF numerically "greater than" NYC, which means nothing — a model that uses distance or magnitude will treat that fake ordering as real information.

2. One-hot encoding avoids false order, at the cost of width

   Three categories become three binary columns, each independent. No fake ordering, but a column per category — fine for a handful of cities, unworkable for a column with thousands of distinct values.

3. Target encoding is compact, and dangerous by default

   Compressing a category down to "the average outcome for rows like this" packs in real signal in one column — but if that average is computed using the row's own label, the row is handed a feature derived from the very thing you're trying to predict.

Look at the SF row: it's the only row with that city, so naive target encoding's "average outcome for SF" is an average of exactly one number — its own label. The encoded value and the true label become the same number. Leave-one-out encoding excludes the current row before averaging, so a singleton category has no other rows left to average — it falls back to the global mean instead of leaking.

Six rows: NYC,1 LA,0 NYC,1 SF,1 LA,0 NYC,0 — global mean label $= 3/6 = 0.5$:

1. NYC appears three times: 1, 1, 0

   Naive target encoding for every NYC row is the same number, $\tfrac{1+1+0}{3} = 0.667$ — safe, since no single row dominates the average.

2. SF appears exactly once: label 1

   Naive target encoding averages that one label with itself: $\tfrac{1}{1} = 1$ — identical to the row's own label. The "feature" is the answer, restated.

3. Leave-one-out catches it

   • For the SF row, leave-one-out excludes the row itself — and there are no other SF rows to average, so it falls back to the global mean, $0.5$.
   • For an NYC row, e.g. the one labeled $0$, leave-one-out averages the other two NYC labels ($1, 1$), giving $1.0$ — a real average of other rows, not a copy of its own answer.