The new server sat almost empty for six hours
Every deploy leaves the fleet lopsided. A replaced server comes back holding nothing, the ones that were not replaced stay at capacity, and the imbalance persists into the next working day. The balancer is configured for round robin and is doing exactly what it was asked.
Meanwhile the deploy itself takes forty minutes, because the runbook says to stop sending new connections to a server and wait for the existing ones to finish. They do not finish. That is what they are for.
- R1A new connection must go to whichever server is currently holding the fewest, rather than to whichever is next in turn. Turn-taking balances arrivals, and arrivals only balance load when things leave as fast as they arrive.
- R2That decision needs a live count of what each server is actually holding, published by the servers themselves and current within seconds. A count that is stale by a minute sends every connection made in that minute to the same place.
- R3Taking a server out of service means ending its connections deliberately and spread over time, because they will not end on their own. Ending them all at once returns every one of those clients to the fleet in the same instant.
Outside every boundary: Arriving clients, Server A (full, untouched), Server B (just replaced, empty), an empty slot for the sends a new connection somewhere, tier 1, an empty slot for the what each server is holding, now, tier 1, an empty slot for the empties a server on purpose, tier 2 Connections: Arriving clients calls sends a new connection somewhere sends a new connection somewhere holds an open connection to Server A sends a new connection somewhere holds an open connection to Server B what each server is holding, now controls sends a new connection somewhere — who is holding what Server A controls what each server is holding, now empties a server on purpose controls Server B — close, slowly