wtf( )unctionsystem design, drawn
← all problemsObservabilityHard

They kept one percent of traces, and one percent of the errors

Tracing was costing more than the service it watched, so the fleet was set to keep one request in a hundred. The bill came down and so did the usefulness: during the next incident there were four traces of the failing path, out of nine hundred failures.

The sampler was not misconfigured. It was asked to decide at the start of each request, and at the start of a request nobody knows whether it is going to fail.

  1. R1Whether a trace is kept must be able to depend on how it turned out — an error, or unusual slowness. That means the choice happens once the trace is complete, not when its first span is created.
  2. R2Every span belonging to one trace must arrive at the same place, or there is no complete trace to judge. Spans of a trace spread across collectors leave each holding a fragment, and a fragment cannot be examined for an error that happened in a different fragment.
  3. R3Spans wait in memory until their trace finishes or a deadline passes. This is a buffer sized for traffic rather than a record: losing it costs some traces and must not cost correctness anywhere, and it must not be something whose growth is unbounded.
Compose the sampling path. Tier 1 is where the spans of one trace meet and what holds them while it waits, tier 2 is what chooses.
Components — tap one, then tap a slot on the diagram
?A decision taken at the beginning cannot depend on the ending. That is not a bug to fix in the sampler; it is the shape of deciding early.

Outside every boundary: Service A (starts the trace), Trace storage (what you pay for), Service B (where it fails; FAILED: errors here), an empty slot for the sends one trace's spans together, tier 1, an empty slot for the where spans wait for the ending, tier 1, an empty slot for the keeps the ones worth keeping, tier 2 Connections: Service A publishes to sends one trace's spans together Service B publishes to sends one trace's spans together sends one trace's spans together calls where spans wait for the ending — same trace, same place where spans wait for the ending calls keeps the ones worth keeping — when it completes keeps the ones worth keeping publishes to Trace storage — the kept ones

Service Astarts the trace
Trace storagewhat you pay for
Service Bwhere it failserrors here