Model monitoring, drift detection & canary rollouts

Hook

A dashboard that only tracks accuracy will tell you a model is fine right up until the moment it isn't. That's not a bug in the dashboard — it's a real blind spot. Accuracy can average out a distribution shift so completely that the two numbers are identical to the decimal.

Intuition

low

0.20

medium

0.50

high

0.30

Overall accuracy: 0.840

Drift score vs. training: 0.0000 bits

No drift detected

Switch snapshots. The bars visibly reshape — traffic that used to sit mostly in the medium bin now spreads across all three — while the accuracy readout barely twitches.

Formalize

Split inputs into bins, each with its own fixed accuracy $a_i$ . Overall accuracy is those accuracies weighted by how often each bin occurs:

\text{accuracy} = \sum_i P(\text{bin}_i)\, a_i

$\text{accuracy}$ — the overall model accuracy on live traffic, averaged across bins.
$i$ — index ranging over each bin the inputs are split into.
$P(\text{bin}_i)$ — the fraction of inputs that fall into bin $i$ .
$a_i$ — the fixed accuracy the model gets on inputs in bin $i$ .

A drift score measures the distributions directly, independent of what accuracy happens to do:

\text{drift} = D_{KL}(P_{\text{train}} \,\|\, P_{\text{live}})

$\text{drift}$ — the drift score: how far the live input distribution has moved from the training one.
$D_{KL}(\cdot \Vert \cdot)$ — the KL divergence, short for Kullback–Leibler divergence: the extra bits it costs to describe one distribution using a code built for the other.
$P_{\text{train}}$ — the distribution of inputs the model was trained and validated on.
$P_{\text{live}}$ — the distribution of inputs the model is actually seeing in production right now.

Accuracy can hide real change
Two very different bin distributions can land on the exact same weighted sum in the accuracy formula above — a coincidence of arithmetic, not evidence that nothing changed.
Drift alerts independent of accuracy
Above a chosen threshold, the input distribution has moved enough to alert on — regardless of whether accuracy noticed.

Play

snapshotaccuracydrift vs. training

training0.8400.0000 bits

live0.8400.1270 bits

An accuracy-only dashboard would show a flat line at 0.84 across both snapshots. Nothing in that number says the input distribution moved at all.

Same accuracy, four significant digits deep. A completely different drift score. One of these two numbers is actually watching the thing that changed.

Worked example

Training distribution: 20% low, 50% medium, 30% high
Weighted accuracy: $0.2(0.95) + 0.5(0.85) + 0.3(0.75) = 0.19 + 0.425 + 0.225 = 0.84$ .
Live distribution: 30% low, 30% medium, 40% high
Weighted accuracy: $0.3(0.95) + 0.3(0.85) + 0.4(0.75) = 0.285 + 0.255 + 0.3 = 0.84$ — identical, to the hundredth.
The distributions are not identical
$D_{KL}(P_{\text{train}} \| P_{\text{live}}) = \sum_i P_{\text{train}}(i)\log_2\frac{P_{\text{train}}(i)}{P_{\text{live}}(i)}$ :
- Low: $0.2\log_2(0.2/0.3) \approx 0.2(-0.585) \approx -0.117$
- Medium: $0.5\log_2(0.5/0.3) \approx 0.5(0.737) \approx 0.368$
- High: $0.3\log_2(0.3/0.4) \approx 0.3(-0.415) \approx -0.125$
Sum: $-0.117+0.368-0.125\approx0.127$ bits — well past a 0.05-bit alert threshold.
- The medium bin's share fell from 50% to 30% (a 40% relative drop).
- The high bin's share rose from 30% to 40% (a one-third relative gain).
Accuracy never had a chance to see it, because the per-bin accuracies happened to average out.

Checkpoint

Three candidate live distributions. Find the one whose accuracy looks unchanged from training (≈0.84) — the one an accuracy-only dashboard would wave through — but whose distribution has genuinely drifted past the 0.05-bit alert threshold.

Pick a candidate to try it

Summary

\text{accuracy} = \sum_i P(\text{bin}_i)\, a_i \qquad \text{drift} = D_{KL}(P_{\text{train}} \,\|\, P_{\text{live}})

Accuracy is a lossy summary — it compresses a whole distribution of behavior into one number, and lossy compression can hide exactly the kind of change that matters. A drift score doesn't replace accuracy monitoring; it watches something accuracy structurally cannot: whether the inputs still look like the ones the model was validated on. Real feature stores and monitoring systems track both, because either one alone eventually gets fooled. The next chapter turns from watching a single model in production to training one across many machines at once.