wtf( )unctionsystem design, drawn
← all problemsML EngineeringMedium

Which plane a scoring request is allowed to touch

Checkout calls a model for a risk score and has 40 ms for the whole answer. The features it needs — how many orders this account placed in the last hour, its lifetime chargeback rate — are computed by a job that runs on a schedule and lands in two places.

Someone wired the lookup at the box with all the history in it, which is the one place in this diagram that has never been able to answer in 40 ms.

Tap the components in the order a scoring request should travel them.
!p99 went from 38 ms to 1.8 seconds. The feature lookup is querying the warehouse.

Boundaries, outermost first: Request path: Online store (one key, one read), Scoring API (40 ms budget), Checkout (asks for a score), Model server (one forward pass), Decision policy (score to a verdict) Outside every boundary: Warehouse (months of history; FAILED: 1.4 s per query), Feature job (runs hourly) Connections: Checkout calls Scoring API Scoring API calls Online store — one key lookup Online store calls Model server — feature vector Model server calls Decision policy — score Feature job publishes to Warehouse — writes history Warehouse publishes to Online store — materialised Scoring API calls Warehouse — what it does today

Tap the components in the order a request travels them.