One of these may be thrown away and one may not
A source-control and CI product installed on infrastructure the customer owns. It has several different kinds of state, and the install guide the team wrote lists them as one line: "a key-value store".
Last month the store filled up, evicted what it needed to, and the product recovered instantly — except that every logged-in user was signed out and a day of queued background jobs had gone. Nothing failed. The eviction policy did exactly what an eviction policy does.
- R1Rendered fragments and computed lookups are kept only to avoid recomputing them. Losing all of it costs time and nothing else, and whatever holds it is expected to discard entries when it runs out of room.
- R2Sessions and the queue of background jobs must survive a restart and must never be discarded to make room. An entry disappearing here is a signed-out user or a job that will not run, and no eviction policy may be able to choose it.
- R3These two are separate stores, not one store used twice. They must not share a failure domain, a memory limit, or an eviction policy — a single instance serving both means pressure from the disposable half decides what happens to the half that matters.
- R4Artifacts, uploads and backups are large and binary. They belong somewhere addressed by name and priced for bulk rather than in either of the above.
Outside every boundary: Job runners (consume the queue), Web nodes (stateless), Product database (primary and replicas), an empty slot for the recomputable, discarded under pressure, tier 1, an empty slot for the sessions and queued work, tier 1, an empty slot for the large, binary, priced for bulk, tier 2 Connections: Web nodes calls recomputable, discarded under pressure Web nodes calls sessions and queued work Job runners calls sessions and queued work — takes work Web nodes calls Product database Web nodes calls large, binary, priced for bulk Job runners calls large, binary, priced for bulk