The latency graph was flat, and the spike came round like a clock
A zone lookup: given somebody's coordinates, which market are they in, is this an airport with its own handling rules, which pricing region applies. It runs before almost everything else the company does, on hundreds of thousands of requests a second, and its latency is added to every product.
The averages were healthy for months. Then somebody plotted the p99 against the clock and found a wall of it, arriving every five minutes on the minute, for about two seconds — the length of time it takes to rebuild an index while people are reading it.
- R1This lookup sits on the path of essentially every request in the company, so its latency is added to every product. The p99 must stay under a tenth of a second and in practice far below it — including at whatever moment the data is being brought up to date.
- R2Any instance must be able to answer any query: the question carries no user, key or region to route by, and two requests a second apart reaching two different instances must not disagree. The dataset is small and a few minutes of staleness is acceptable.
- R3A restarting or newly deployed instance must be answering correctly within seconds. The definitions live in several systems owned by other teams — this service owns none of them — and a rolling deploy must not arrive at those teams as a stampede.
Outside every boundary: Other teams' systems (own the answers), Load balancer (any instance), Every app (before anything), an empty slot for the what an instance holds, tier 1, an empty slot for the how it takes an update, tier 1, an empty slot for the what it serves from at start, tier 2 Connections: Every app calls Load balancer — which zone? Load balancer calls what an instance holds — any of them how it takes an update calls what an instance holds — brings it up to date how it takes an update calls Other teams' systems — polls how it takes an update publishes to what it serves from at start — writes what it serves from at start calls what an instance holds — read at start