Part VI — Unsupervised Learning, Clustering, Dimensionality & Time Series · Chapter 7

Anomaly detection: Isolation Forests & LOF

Hook

Every classifier so far needed labeled examples of what "wrong" looks like. What finds the one weird point in a dataset when nobody has ever labeled anything as an outlier at all?

Intuition

Six points: five clustered close together, one sitting far off on its own. No labels, no training signal about which one is "the anomaly" — just an anomaly score for each point, computed from nothing but how the data happens to be spread out.

Formalize

An isolation tree repeatedly splits a dataset, at each step separating it into two groups, until every point sits alone. The score is just how many splits that took:

anomaly(x)1depth(x)\text{anomaly}(x) \propto \frac{1}{\text{depth}(x)}
  • anomaly(x)\text{anomaly}(x) — the anomaly score assigned to point xx.
  • xx — a single data point being scored.
  • depth(x)\text{depth}(x) — how many splits an isolation tree needs before xx ends up alone.
  1. Outliers isolate almost immediately

    A point sitting off on its own tends to land alone after just one or two splits — there's nothing else nearby to keep splitting away from it.

  2. Clustered points need many more splits

    A point buried in a dense cluster needs several splits before every one of its close neighbors has been separated out too.

Play

The outlier isolates in a single split. Every clustered point needs three or four. That gap is the entire signal isolation forests use — no notion of "normal," no boundary to fit, just: how fast does a random cut leave this point by itself?

Worked example

Points (1,2,3,4,5,20)(1, 2, 3, 4, 5, 20), split at the midpoint of whatever range remains each time:

  1. The very first split isolates the outlier

    Range [1,20][1, 20] splits at its midpoint, 10.510.5. Everything below goes one way, {20}\{20\} goes the other, alone — isolated after exactly 11 split.

  2. The cluster needs several more splits
    • {1,2,3,4,5}\{1,2,3,4,5\} splits at 33, giving {1,2}\{1,2\} and {3,4,5}\{3,4,5\}
    • Splitting {1,2}\{1,2\} again at 1.51.5 isolates both 11 and 22, at depth 33
    • Splitting {3,4,5}\{3,4,5\} at 44 isolates 33, also at depth 33
  3. The last two take one split further

    {4,5}\{4,5\} needs one more split, at 4.54.5, before 44 and 55 are each isolated — at depth 44, the deepest of any point in this dataset.

Checkpoint

Click the point that isolates in the fewest splits — the anomaly.

Click a point to try it
Summary
anomaly(x)1depth(x)\text{anomaly}(x) \propto \frac{1}{\text{depth}(x)}

A real isolation forest builds many trees, each picking a random feature and a random split point rather than always cutting at the midpoint, then averages the isolation depth across all of them — smoothing out the luck of any one particular tree. But the core mechanism is exactly what showed up here: points that are rare and different tend to fall away from the rest of the data almost immediately, and that's a signal worth measuring even when nobody has ever told the model what "anomalous" means.