wtf( )unctionsystem design, drawn
← all problemsDeploymentHard

The zero-downtime deploy that dropped requests

The rollout replaces instances one at a time and waits for each new one to pass its health check before continuing. That is the definition of a zero-downtime deploy and it is configured correctly.

Health checks govern when traffic starts. Nothing here says anything about how it stops. The instance being replaced is still registered, still holding a forty-second request, and is killed thirty seconds after being told to go.

Order the shutdown so the instance leaves rotation before it stops serving, and give it long enough to finish.
Components — tap one, then tap a slot on the diagram
!Every rolling deploy produces a burst of 5xx. Health checks are green from start to finish.

Outside every boundary: A 40-second request (still running), Orchestrator (grace period 30s; FAILED: shortest deadline), Load balancer (drain window 300s), Instance being replaced (still registered; FAILED: killed mid-request), an empty slot for the what the instance does first, an empty slot for the the deadline that outlasts the others Connections: Load balancer calls Instance being replaced — still sending (step 1) what the instance does first controls Load balancer — take me out (step 2) A 40-second request calls Instance being replaced — 40s of work left the deadline that outlasts the others controls Orchestrator (step 3) Orchestrator must NOT reach Instance being replaced — killed at 30s

A 40-second requeststill running
Orchestratorgrace period 30sshortest deadline
Load balancerdrain window 300s
Instance being replacedstill registeredkilled mid-request