Adding the fifth shard moved everything
Four shards, keys placed by taking the remainder of a hash. It spreads evenly, it has been fine for two years, and the plan to add a fifth shard was written down as a configuration change.
It is not one. Going from four to five changes where almost every key belongs, so the "configuration change" is a migration of the entire dataset — and separately, one customer has grown large enough that no placement function will help, because their traffic does not fit on a shard however cleverly it is chosen.
- R1Adding a shard must move only a fraction of the existing keys. A scheme that changes where almost every key belongs is a full data migration wearing the costume of a config change, and it has to happen while the system is serving traffic.
- R2A small number of keys must be placeable by hand, overriding whatever the function says, without changing the function or moving anything else. That decision must survive a restart and be answerable for any key.
- R3One customer's reads now exceed what a single shard can serve, and no choice of placement fixes that — their data is one key's worth. It must be servable from more than one copy, at the cost of those copies being slightly behind.
Outside every boundary: Shard B (holds the big customer), Shard A, Application (asks by key), an empty slot for the turns a key into a shard, tier 1, an empty slot for the where a hand-placed key is recorded, tier 1, an empty slot for the serves one key more than once, tier 2 Connections: Application calls turns a key into a shard turns a key into a shard controls where a hand-placed key is recorded — any exceptions turns a key into a shard calls Shard A turns a key into a shard calls Shard B Shard B publishes to serves one key more than once — replicates Application calls serves one key more than once — big customer reads