Build and deploy a monitored, high-throughput AI serving cluster

Hook

Every piece of this part has been built and tested on its own — the feature join, the accuracy dashboard, the drift alarm. Wire all three into one pipeline and watch what a production incident actually looks like: every static test green, the dashboard calm, and a real problem happening anyway.

Intuition

low

0.30

medium

0.40

high

0.30

Ingestion: static join tests 3/3 passing

Serving: accuracy = 0.800

Monitoring: drift = 0.0000 bits (threshold 0.05)

All clear

Step through three days. The ingestion tests never change — they're checking code, not traffic. Watch for the day where accuracy agrees with day one, but the drift monitor doesn't.

Formalize

A monitored serving pipeline is three independent checks, each blind to what the others catch:

\underbrace{\text{ingestion}(t)}_{\text{Ch. 1}} \quad \underbrace{\text{accuracy} = \sum_i P(\text{bin}_i)\,a_i}_{\text{Ch. 3}} \quad \underbrace{D_{KL}(P_{\text{train}} \,\|\, P_{\text{live}})}_{\text{Ch. 3}}

$\text{ingestion}(t)$ — the static test suite's check: does the feature join still return the correct, point-in-time value for a fixed historical label at time $t$ ?
$\text{accuracy}$ — the weighted-average accuracy across bins of live traffic, from the model-monitoring chapter.
$P(\text{bin}_i)$ — the fraction of live inputs falling into bin $i$ .
$a_i$ — the fixed accuracy within bin $i$ .
$D_{KL}(P_{\text{train}} \Vert P_{\text{live}})$ — the drift score: the KL divergence, short for Kullback–Leibler divergence, between the training and live input distributions.

Static tests verify code, not traffic
A static test suite checks that the join still returns the right number for a fixed historical label — today, tomorrow, forever.
That says nothing about live traffic
A green test suite is silent on what today's live traffic actually looks like, because it never queries live traffic in the first place.
Only a distribution-level monitor watches that
$D_{KL}(P_{\text{train}} \Vert P_{\text{live}})$ is designed to watch exactly the thing accuracy structurally can't — the shape of the incoming distribution, independent of whether it changes the accuracy number.

Play

daystatic testsaccuracydriftalarm

1passing0.8000.0000no

2passing0.8050.0103no

3passing0.8000.1510yes

Day 3's accuracy is identical to day 1's. A dashboard watching accuracy — or a CI suite checking the join logic — would report nothing wrong on any of these three days.

Three static tests pass on all three days — the ingestion code never changed. Accuracy holds at 0.800 on day 1 and day 3 alike. The only signal that anything happened on day 3 is the drift score, because it's the only one of the three checks actually looking at the shape of the traffic.

Worked example

Day 1: the baseline
- Distribution matches training exactly: $[0.3,0.4,0.3]$ .
- Accuracy $=0.3(0.9)+0.4(0.8)+0.3(0.7)=0.27+0.32+0.21=0.800$ .
- Drift $=D_{KL}([0.3,0.4,0.3]\|[0.3,0.4,0.3])=0.000$ bits, since the two distributions are identical.
Every check is green because nothing has happened yet.
Day 3: the incident
Distribution shifts from $[0.3,0.4,0.3]$ (train) to $[0.4,0.2,0.4]$ (live):
- Medium-bin traffic falls from 40% to 20% (halved).
- High-bin traffic rises from 30% to 40% (up by a third).
- Accuracy: $0.4(0.9) + 0.2(0.8) + 0.4(0.7) = 0.36 + 0.16 + 0.28 = 0.80$ — unchanged, because the bins' fixed accuracies happened to average out identically.
What actually caught it
$D_{KL}([0.3,0.4,0.3]\|[0.4,0.2,0.4])$ :
- $0.3\log_2(0.3/0.4) \approx 0.3(-0.415) \approx -0.125$
- $0.4\log_2(0.4/0.2) = 0.4\log_2(2) = 0.4(1) = 0.4$
- $0.3\log_2(0.3/0.4) \approx -0.125$
Sum: $-0.125+0.4-0.125\approx0.151$ bits, well past the 0.05 threshold. Not the static ingestion tests — they check a join function against fixed labels, not live traffic. Not the accuracy dashboard — it produced the same number as a day with no problem at all. Only the piece built specifically to compare distributions caught anything.

Checkpoint

Three new candidate days. Find the one whose accuracy looks completely normal — the static tests are green, the accuracy dashboard is flat — but that a drift alarm should still catch, exactly like day 3.

Pick a candidate to try it

Summary

\text{ingestion}(t) \quad \text{accuracy} = \sum_i P(\text{bin}_i)\,a_i \quad D_{KL}(P_{\text{train}} \,\|\, P_{\text{live}})

None of these three checks is redundant with the others, and none of them can substitute for the others — that's the actual argument for building a monitored pipeline instead of trusting any single metric. A static test suite catches code regressions. An accuracy dashboard catches obvious performance drops. A drift alarm catches the distribution shifting under a model that still looks, by every other measure, fine. Real production incidents live in the gaps between whichever subset of these a team happened to build. This closes Part XIX and the curriculum's tour of what happens to a model after training ends.