Scaling that arrives after the traffic
Inference runs on GPU nodes that autoscale on CPU utilisation. Under load the GPUs saturate while CPU sits at 30%, so nothing scales — and when a node finally is added, it spends four minutes pulling a 30GB model before serving anything.
By the time capacity arrives the spike is over. Then the queue drains, utilisation drops, and the scaler removes the node you just paid to warm.
Boundaries, outermost first: Inference fleet: GPU node (serving), GPU node (cold; FAILED: cold start) Outside every boundary: Request queue, Requests, an empty slot for the a signal that tracks load, an empty slot for the shorten the cold start Connections: Requests calls Request queue (step 1) Request queue controls a signal that tracks load — the real signal (step 2) a signal that tracks load controls GPU node — scale out (step 3) shorten the cold start calls GPU node — ready in seconds (step 4)