The containers that mattered had already exited
A multi-tenant platform runs a few hundred short-lived containers. Its logging works right up until it is needed.
During the last incident the containers worth reading had already been replaced, and their lines went with them. The ones still running fell over when the log backend slowed down — not because the backend was down, but because the applications were waiting on it. Somewhere in the middle of it, an engineer searching for one tenant's errors was shown another tenant's.
- R1A container may be terminated at any moment and anything it has already written must survive that, so nothing may be held only inside the process that produced it. The destination must also be changeable without rebuilding or reconfiguring the application — which rules out the same designs for the same underlying reason.
- R2When the destination is slow or briefly unavailable the applications must not slow down AND must not drop what they emit. Both halves are stated because the easy fix trades one for the other: log delivery must never be able to take the platform down, and must never be the reason a line is missing.
- R3Engineers query the last thirty days constantly and interactively. Compliance requires seven years of the same data and reads it a handful of times a year. Those are two different products — something you ask questions of, and bytes you can produce on demand — and sizing one store for both pays archive prices for query speed nobody uses.
- R4Every line must carry which tenant produced it, stamped by something that runs over each line rather than by the application choosing to include it.
- R5A tenant must never see another tenant's lines. Labelling a line and refusing to return somebody else's are two different jobs, and the component that does the second has to decide, not merely annotate.
Outside every boundary: On call (searching now), Tenant app (replaced hourly), Tenant app (one of ~200), Compliance (twice a year), an empty slot for the takes lines out of a dying container, tier 1, an empty slot for the absorbs a burst the sink cannot, tier 2, an empty slot for the stamps whose line this is, tier 2, an empty slot for the thirty days, queried all day, tier 3, an empty slot for the seven years, read twice a year, tier 3, an empty slot for the shows a tenant only their own, tier 3 Connections: Tenant app controls takes lines out of a dying container Tenant app controls takes lines out of a dying container takes lines out of a dying container publishes to absorbs a burst the sink cannot absorbs a burst the sink cannot publishes to stamps whose line this is stamps whose line this is publishes to thirty days, queried all day stamps whose line this is publishes to seven years, read twice a year thirty days, queried all day calls shows a tenant only their own shows a tenant only their own calls On call seven years, read twice a year calls Compliance