Two leaders, both certain
A nightly billing job must run on exactly one node, guarded by a lock with a thirty-second lease. Node A takes the lock, then pauses for forty-five seconds — garbage collection, a stalled disk, anything. The lease expires, Node B takes it and starts writing. Node A wakes up still believing it holds the lock, and writes too.
Every customer is billed twice. No lock with a timeout can prevent this, because a paused process cannot notice that it paused.
Outside every boundary: Node B (holds the lease), Node A (paused, lease expired; FAILED: stale writer), Billing store, Lock service (issues leases), an empty slot for the checked on every write Connections: Lock service controls Node B — lease + token 34 (step 1) Node B calls Billing store — writes with 34 (step 2) Node A must NOT reach Billing store — token 33 — rejected (step 3) checked on every write controls Billing store — enforced here