The invalidation arrived before the data did
A read-dominated social graph, cached in every region. Writes go to one region and replicate outward; the cache in each region has to be told when something it holds has changed.
The first version sent those notices over their own fast channel and produced a bug nobody could reproduce: a cached value that was wrong, stayed wrong, and had been invalidated correctly.
- R1Application code talks only to a nearby cache tier and never to the database or to whatever fronts it. That tier holds values it is willing to lose at any moment and answers most reads from them.
- R2Exactly one thing per shard reads from and writes to the database, and it is the only thing that does. Two of them would issue conflicting refills for the same key and each would believe its own.
- R3A notice that a value has changed must never reach a region before the change itself has. It has to travel in the same ordered stream as the data, not alongside it — a separate channel is faster, and faster is the failure.
Boundaries, outermost first: THIS REGION: Application (reads constantly), Local replica (applies the stream), an empty slot for the what the application is allowed to ask, tier 1, an empty slot for the the only thing that touches the store, tier 1 WRITE REGION: Primary (takes the writes), an empty slot for the carries changed-value notices outward, tier 2 Connections: Application calls what the application is allowed to ask what the application is allowed to ask calls the only thing that touches the store — miss, or write the only thing that touches the store calls Local replica Primary publishes to carries changed-value notices outward carries changed-value notices outward publishes to Local replica — with the data carries changed-value notices outward controls the only thing that touches the store — changed