wtf( )unctionsystem design, drawn
← all problemsResilienceHard

Four days long, and the vendor may never call you

An onboarding process runs for four days. It calls three third parties, waits twice for a human to approve something, and sleeps for long stretches in between. Compliance has agreed the orchestration can be bought rather than built — on one condition: the code that touches customer data and holds the third-party credentials runs inside the company's own network, and nothing outside may open a connection into it.

The vendor's own quickstart begins by asking for a callback URL.

  1. R1The orchestration is bought and runs outside the company's network. The workers that execute each step run inside it, and nothing outside may open a connection inward — no callback, no webhook, no push. Whatever links them must therefore be dialled from inside and never dialled from outside.
  2. R2A sleep measured in days must fire, and the application is redeployed about twenty times a day. Anything holding a pending timer in the memory of a process that restarts is not holding it, and a process that has been asleep for three days has no way to notice it was forgotten.
  3. R3An operator must be able to ask, of any process still in flight, where it has got to and why it is stuck — by asking a question, not by reading application logs and not by replaying an execution's history to find out where it stopped.
Compose the parts the vendor cannot supply. Tier 1 is how the workers and the vendor talk and what makes a four-day sleep survive, tier 2 is how an operator finds a stuck process.
Components — tap one, then tap a slot on the diagram
?Every requirement here is satisfiable except in combination. The vendor runs outside, the code runs inside, and nobody outside may dial in — which decides who initiates, and that decides everything else.

Boundaries, outermost first: OUR NETWORK: Credentials (never leave here), Step worker (runs our code) Outside every boundary: Operator (asks what is stuck), Orchestrator (the vendor's, outside), an empty slot for the how the workers reach the vendor, tier 1, an empty slot for the holds a four-day sleep, tier 1, an empty slot for the which processes are stuck, tier 2 Connections: Step worker calls how the workers reach the vendor how the workers reach the vendor sends outbound traffic to Orchestrator — always dialled out Step worker calls Credentials Orchestrator controls holds a four-day sleep Orchestrator publishes to which processes are stuck which processes are stuck calls Operator

Credentialsnever leave here
Operatorasks what is stuck
Step workerruns our code
Orchestratorthe vendor's, outside