They kept one percent of traces, and one percent of the errors
Tracing was costing more than the service it watched, so the fleet was set to keep one request in a hundred. The bill came down and so did the usefulness: during the next incident there were four traces of the failing path, out of nine hundred failures.
The sampler was not misconfigured. It was asked to decide at the start of each request, and at the start of a request nobody knows whether it is going to fail.
- R1Whether a trace is kept must be able to depend on how it turned out — an error, or unusual slowness. That means the choice happens once the trace is complete, not when its first span is created.
- R2Every span belonging to one trace must arrive at the same place, or there is no complete trace to judge. Spans of a trace spread across collectors leave each holding a fragment, and a fragment cannot be examined for an error that happened in a different fragment.
- R3Spans wait in memory until their trace finishes or a deadline passes. This is a buffer sized for traffic rather than a record: losing it costs some traces and must not cost correctness anywhere, and it must not be something whose growth is unbounded.
Outside every boundary: Service A (starts the trace), Trace storage (what you pay for), Service B (where it fails; FAILED: errors here), an empty slot for the sends one trace's spans together, tier 1, an empty slot for the where spans wait for the ending, tier 1, an empty slot for the keeps the ones worth keeping, tier 2 Connections: Service A publishes to sends one trace's spans together Service B publishes to sends one trace's spans together sends one trace's spans together calls where spans wait for the ending — same trace, same place where spans wait for the ending calls keeps the ones worth keeping — when it completes keeps the ones worth keeping publishes to Trace storage — the kept ones