A node upgrade took auth down
A routine node-pool upgrade drained nodes in parallel. All three replicas of the auth service happened to be on two nodes in one zone, so they were evicted within seconds of each other and auth was down for ninety seconds.
The deployment had three replicas and a rolling-update strategy, and everyone assumed that covered it. Rolling-update settings govern *your* rollouts — they have nothing to say about someone else draining a node.
Boundaries, outermost first: Zone A: auth pod (FAILED: evicted), auth pod (FAILED: evicted) Zone B: auth pod Outside every boundary: Node drain (upgrade in progress), an empty slot for the how many may go at once, an empty slot for the where they're placed Connections: Node drain controls how many may go at once — eviction request (step 1) how many may go at once controls auth pod — one at a time (step 2) where they're placed controls auth pod — one per zone