They added a cache, and the thousand queries arrived anyway
Community chat, trillions of stored messages, rooms ranging from five friends to several hundred thousand members. The dominant read is always the same shape: the newest messages in one room, then the page before that.
Something notable happens in one large room. Everyone opens it inside the same second, all asking for the same newest page. The team had put a read cache in front of the store for exactly this, and the store still took the full weight of it — then the rooms that had nothing to do with any of it started timing out too.
- R1A very large number of members open the same room in the same second and ask for the same newest page. The storage layer must not see that as a proportional increase in queries. A cache is not by itself an answer here: the spike arrives on a page nobody has read yet, and every one of those callers misses.
- R2Requests that could be answered by one piece of work have to actually meet. Two callers asking the identical question of two different instances are two questions, however clever each instance is on its own.
- R3Tail latency is the product; a one-second stall is a visible failure. A stall caused by one busy room must not degrade rooms that have nothing to do with it and happen to share the hardware. Shedding the excess is not an answer either — a refused request is a failure the member can see.
Boundaries, outermost first: THE READ PATH: API servers (one call per open), an empty slot for the what one room may hold, tier 1, an empty slot for the how callers reach the tier, tier 1, an empty slot for the when they all want one thing, tier 2 Outside every boundary: Message store (by room and window), Members (opening the room), Three copies (of every partition) Connections: Members calls API servers — open the room API servers calls what one room may hold — the newest page what one room may hold calls how callers reach the tier — what got through how callers reach the tier calls when they all want one thing — by room when they all want one thing calls Message store — the query Message store publishes to Three copies — replicated