Review this evaluation setup
An assistant team built a scoreboard so prompt and model changes stop shipping on vibes. There is a seed set a support lead graded by hand, a sampler that captures a slice of production answers, a set of cases the candidate is scored on, a model that does the scoring, and a gate that blocks anything scoring worse than what is live.
It runs on every pull request and it has never once blocked a change.
Two components on this diagram are wired to the wrong thing. The rest are fine and are there to be checked rather than circled.
Outside every boundary: Judge model (the candidate model), Hand-graded seed (40 cases, by a lead), Promotion gate (blocks a regression), Trace sampler (1% of answers), Evaluation set (grown from those), Real questions (production traffic), Candidate (up for promotion), Model in production (v12, the incumbent) Connections: Real questions calls Model in production Model in production publishes to Trace sampler — answers it gave Trace sampler publishes to Evaluation set — become new cases Hand-graded seed controls Evaluation set — the original 40 Candidate calls Judge model — same weights Evaluation set calls Judge model — the cases Judge model controls Promotion gate — a score Promotion gate calls Model in production — promotes