wtf( )unctionsystem design, drawn
← all problemsML EngineeringMedium

Monitoring that cannot fire for another two months

A credit model decides who gets an instant approval. Whether the decision was right — did the account actually default — is known 60 to 90 days later, so the team's monitoring computes accuracy against those outcomes.

An upstream service started sending income in cents on Tuesday. Every applicant now looks a hundred times richer than they are. The accuracy dashboard for this quarter is blank, because the labels for it do not exist yet.

Place the check that can fire today, on a signal that needs no labels at all.
Components — tap one, then tap a slot on the diagram
!Income has arrived in cents since Tuesday. Every monitoring dashboard is green.

Boundaries, outermost first: Serving path: Applications (live traffic), Risk model (in production; FAILED: income now in cents), Scoring service, Decision (approve or decline) Outside every boundary: Served features (what the model saw), Outcomes (60-90 days later), Accuracy monitor (needs labels; FAILED: nothing to score yet), an empty slot for the fires without labels Connections: Applications calls Scoring service (step 1) Scoring service calls Risk model — feature vector (step 2) Risk model calls Decision (step 3) Scoring service controls Served features (step 2) Served features calls fires without labels — against the training window (step 4) Decision publishes to Outcomes (step 5) Outcomes calls Accuracy monitor (step 6)

Served featureswhat the model saw
Applicationslive traffic
Risk modelin productionincome now in cents
Scoring service
Outcomes60-90 days later
Accuracy monitorneeds labelsnothing to score yet
Decisionapprove or decline