Ten thousand things changed at once and nothing slowed down
A large fleet of request-serving nodes needs to know the current health of everything they might route to. When a whole failure domain goes dark, tens of thousands of entries change status in the same instant — which is precisely the moment this must not get slower.
The correct design here looks wasteful on a quiet day. That is not a side effect of it; it is the mechanism.
- R1The amount of work done per cycle must not depend on how many entries changed, how many customers exist, or how many entries are currently bad. The same work happens when nothing at all has changed.
- R2The fleet receives one complete picture rather than a series of changes. A node that misses a cycle recovers on the next one, with no backlog to drain and nobody replaying anything, and a partially applied load corrects itself the same way.
- R3A serving node answers from its own local copy and never calls the assembling side while serving a request, so the assembler being down cannot stop the fleet serving. That local copy is not a second source of truth and must not outlive a restart.
Boundaries, outermost first: SERVING NODE: Answering process (reads its own copy), an empty slot for the what the node answers from, tier 2 Outside every boundary: Requests, Health checkers (same work every cycle), an empty slot for the builds one table, however much moved, tier 1, an empty slot for the one complete picture, fetched whole, tier 1 Connections: Health checkers controls builds one table, however much moved builds one table, however much moved controls one complete picture, fetched whole — written every cycle one complete picture, fetched whole controls what the node answers from — fetched whole what the node answers from calls Answering process Requests calls Answering process