The health check that removed every server
The load balancer's health check queries the database to prove the instance can really serve. During a brief database slowdown every instance failed its check at once, and the load balancer removed the entire fleet from rotation.
Every instance was running. Every instance was healthy. The check was measuring a dependency they all share, so it deregistered all of them simultaneously.
Boundaries, outermost first: Fleet: Instance (FAILED: pulled), Instance (FAILED: pulled), Instance (FAILED: pulled) Outside every boundary: Shared database (briefly slow; FAILED: slow), Load balancer, an empty slot for the when everything looks unhealthy Connections: Load balancer controls when everything looks unhealthy — health results (step 1) when everything looks unhealthy calls Instance (step 2) when everything looks unhealthy calls Instance Instance calls Shared database (step 3)