They saved it, the page reloaded, and the old value was back
A profile service has one database and a complaint that will not go away: people edit their profile, the page reloads, and the old value is back. Saving again usually works.
It is intermittent. It is worse under load. It has never once reproduced on anybody's laptop, and three engineers have now gone looking for a race in the save handler. There is no race in the save handler.
- R1Immediately after someone saves a change, that person's next read must show it. No delay is acceptable — not a short one, not one that is usually fine. Whatever serves this read must be the same thing the write went to.
- R2Everybody else may see the change up to a few seconds late, and they are by far the larger share of the traffic. What serves them must not be the thing serving R1, and must not be something every write has to wait for before it is acknowledged.
- R3A nightly export scans every profile. It must not share a target with the live read path, must not be pointed at the write path, and must not be something a write waits on either.
- R4Profile photos are large, immutable once uploaded, and requested far more often than profile text. They must not be kept by whatever answers profile reads.
- R5Losing an acknowledged save is not acceptable. Losing something whose only job is to make reads faster is entirely acceptable, and the design should say which is which rather than making everything durable.
Outside every boundary: Viewing (everyone else), Profile API (routes the reads), Nightly export (scans every row), Just saved (the same person), Edge cache (where photos are served), an empty slot for the takes writes, and the author reads it, tier 1, an empty slot for the losing it costs latency, not data, tier 2, an empty slot for the a few seconds late is fine here, tier 2, an empty slot for the a full scan that must disturb nobody, tier 2, an empty slot for the large, immutable, written once, tier 3 Connections: Just saved calls Profile API Viewing calls Profile API Profile API calls takes writes, and the author reads it — the author's own read Profile API calls a few seconds late is fine here — everyone else Profile API calls losing it costs latency, not data takes writes, and the author reads it publishes to a few seconds late is fine here — replication takes writes, and the author reads it publishes to a full scan that must disturb nobody Nightly export calls a full scan that must disturb nobody Profile API calls large, immutable, written once large, immutable, written once calls Edge cache Nightly export must NOT reach takes writes, and the author reads it