It scored beautifully offline and disappointed in production
A recommendation model is excellent in evaluation and mediocre live. The team has checked the model twice. The model is fine.
Two things are wrong and neither is in the model. The live call computes the forty signals it needs on the spot, using code written by a different team from the one that wrote the nightly job — so the model is trained on one definition and asked about another. And the training rows were built by joining each past event against the signals as they stand today, which means a row about last Tuesday quietly contains next Wednesday's numbers.
- R1The live call must read its signals rather than compute them. The nightly job and the live call have to obtain each signal from one definition, and the surest way to have two definitions is to have two pieces of code — one written for a batch over history, one written for a request with ten milliseconds to spare.
- R2A training row about a past event must contain only values that were already true at the moment of that event. A signal's value today must never appear in a row about last Tuesday, however convenient the join is to write.
- R3For any recommendation served it must be possible to say afterwards which model version produced it and what it was given. That record is what an incident is reconstructed from, so it must be addable and never rewritable — a record that can be corrected is a record that can be corrected after the fact.
Outside every boundary: Product API (answers in 10ms), Training run (learns from rows), Analyst (asks why it said that), Model server (scores the request), Nightly job (over full history), an empty slot for the forty signals, read not computed, tier 1, an empty slot for the values as they stood that day, tier 1, an empty slot for the what was asked, and what answered, tier 2 Connections: Product API calls Model server Product API calls forty signals, read not computed — forty signals Model server publishes to what was asked, and what answered what was asked, and what answered calls Analyst Nightly job publishes to values as they stood that day values as they stood that day calls Training run values as they stood that day publishes to forty signals, read not computed — published nightly