A model's training set and its serving traffic are supposed to see the same features computed the same way. They usually don't — not because anyone wrote a bug, but because "the same raw event" can honestly produce two different feature values depending on when the pipeline looks at it.
Sum of every purchase that happened strictly before t=5: $45
Same six raw events the whole time — only the question "as of when?" changes the answer.
Six purchases, one user, no ambiguity about what happened or when. Slide the query time and watch the sum change — not because the events changed, but because "as of when?" changed.
A feature store serves the same feature two ways: online, at prediction time, where only the past is visible by construction; and offline, when building a training set from history, where the entire future is sitting right there in the same table unless the pipeline explicitly refuses to look at it.
The correct, point-in-time feature for a label produced at time only uses events strictly before it:
- — the correct, point-in-time feature value: what the feature legitimately looked like as of time .
- — the time a label was produced; the moment a real prediction would have been made.
- — one raw event in the stream (here, a purchase).
- — the timestamp of event .
- — the value contributed by event .
A naive offline join instead grabs whatever the feature table currently holds — the latest snapshot — and attaches it to every historical label, regardless of :
- — the (incorrect) feature value the naive join attaches to a label at time .
- — the current time, when the offline join actually runs — not the label's own time.
- The leak window is every event between t and t_now
Every event that happened after the label's own time but before the offline join actually ran gets folded into regardless — a real prediction made at could never have seen any of it.
The naive join always returns the same $130 — whatever the feature store happens to hold right now — no matter which historical moment the label actually came from.
Three training labels, the same six raw events, two joins. The correct join gives each label a different answer, because each label happened at a different moment. The naive join gives every label the same answer — today's snapshot — because it never consulted the label's own timestamp at all.
- A label lands at t=6
Point-in-time correct: only count purchases strictly before t=6 — the ones at t=1, 2, 4 — for a total of 20 + 45.
- The naive join ignores t=6 entirely
It reaches for the feature table's current total, computed as of t=11 (after every event in the stream): 20+50+30 = $130.
- The gap is the leak
45 = $85 of future spending — purchases at t=6, 8, and 9 — silently informing a feature that's supposed to describe the world before t=6. A model trained on this row learns a correlation that will never exist at serving time, because serving time never has next week's purchases to leak from.
A new label arrives at t=7. Pick the correct point-in-time feature value — the sum of only the purchases that happened strictly before t=7.
The bug never touches the raw events — they're recorded correctly either way. It lives entirely in the join: whether the pipeline asks "what did this feature look like at label time?" or silently substitutes "what does this feature look like right now?" A feature store's whole job is making the first question cheap to ask correctly, every time, instead of leaving it to whichever engineer remembers to filter by timestamp. The next chapter turns to the training runs those correctly-joined features feed into, and what it takes to reproduce one exactly.