wtf( )unctionsystem design, drawn
← all problemsML EngineeringHard

Review this ML platform

A team asks you to look over their platform before it goes in front of an auditor. The training job runs, the model serves, the monitoring is wired up, and every box has an owner.

Three components are connected to the wrong thing. All three work today, in the sense that nothing errors and no alert fires.

Tap every component that is reading from, or writing to, the wrong place.
!0.93 offline, 0.68 live — and nobody can say which weights declined an application last month.

Boundaries, outermost first: Serving plane: Online store (latest per entity), Model server (follows 'current'), Scoring API (public edge) Training plane: Training job (reads live features), Offline store (full history) Outside every boundary: Drift monitor (reads the train set), Event stream (raw activity), Feature job (hourly) Connections: Event stream publishes to Feature job Feature job publishes to Offline store — history Feature job publishes to Online store — current value Online store calls Training job — latest values Training job publishes to Model server — models/current Offline store controls Drift monitor — training rows Scoring API calls Model server — score request

Tap every component that is wrong.