Analytics got slow, and nobody could check out
Analytics is not on the critical path for anything. Checkout calls it at the end of an order to record a funnel event, and the code that does it already ignores the error, because a failed funnel event is not a failed order.
On Thursday analytics went from twenty milliseconds to eight seconds. Checkout then served nothing at all for eleven minutes — including requests that never reach the analytics call. Payments was healthy. Inventory was healthy. Checkout's own process was running the whole time and its health check said so.
- R1While an instance is saturated it must stop receiving new requests. Whatever distributes work across checkout's instances has to choose on what each one is currently holding — a rotation hands a parked instance the same share as an idle one, and a check that the process is running answered yes throughout the incident.
- R2A request that never calls analytics must still be served while analytics is slow. Checkout has a fixed number of workers and an eight-second call holds one for eight seconds, so what must be bounded is the SHARE of checkout's capacity any single dependency may occupy — not how long one call may take, and not how many calls may be in flight altogether, since either of those leaves one slow dependency holding everything.
- R3Awarding points and recording funnel events do not change the answer the customer is given, so they must leave the request path entirely. What they are handed to has to survive their service being down at the moment of the order, and has to survive checkout being redeployed, which happens about twenty times a day.
Outside every boundary: Shoppers (placing orders), Payments (cannot come off the path), Checkout (a fixed pool of workers), Analytics (a funnel event; FAILED: 8 seconds), an empty slot for the chooses which checkout instance gets it, tier 1, an empty slot for the stands between checkout and the calls it must make, tier 1, an empty slot for the takes the work the answer does not depend on, tier 2 Connections: Shoppers calls chooses which checkout instance gets it chooses which checkout instance gets it calls Checkout Checkout calls stands between checkout and the calls it must make stands between checkout and the calls it must make calls Payments Checkout publishes to takes the work the answer does not depend on takes the work the answer does not depend on publishes to Analytics — when it is up again Checkout must NOT reach Analytics — never on the answer's path