The dashboard that went blank during the incident
A release added the customer identifier as a dimension on the request-duration metric, so per-customer latency could be charted. The change is four characters long.
Within a day the metrics store's memory had tripled and queries were timing out. The first time anyone needed those dashboards was during an unrelated outage, and they would not load.
Outside every boundary: Trace store (a record per request), Metrics store (a series per combination; FAILED: out of memory), Service (emits telemetry), an empty slot for the few values, known ahead, an empty slot for the identifies one request Connections: Service controls few values, known ahead — counters (step 1) few values, known ahead controls Metrics store — a few dozen series (step 2) Service controls identifies one request — one span per request (step 3) identifies one request controls Trace store (step 4) Metrics store controls Trace store — exemplar