Ten percent everywhere, and no complete traces
Every service in the chain samples ten percent of traces. That sounds like it should produce ten percent of traces, in full.
It produces about ten percent of spans, scattered. A request is kept by the gateway, dropped by orders, kept again by payments, so the trace that survives has a hole in the middle where the interesting part was. Six services deep, the odds of every hop independently choosing to keep the same request are one in a million.
Boundaries, outermost first: Collector tier: Load balancer (trace-id aware), an empty slot for the where slow-or-failed is judged Outside every boundary: Trace view (parentless spans; FAILED: fragments), Gateway (kept), Payments (kept), Orders (dropped; FAILED: re-decided), an empty slot for the what each hop does with the flag Connections: Gateway calls what each hop does with the flag (step 1) what each hop does with the flag calls Orders — the same decision (step 2) Orders calls Payments (step 3) Payments publishes to Load balancer — spans, as they end (step 4) Load balancer calls where slow-or-failed is judged — same trace, same box (step 5) where slow-or-failed is judged calls Trace view (step 6) Orders must NOT reach Trace view — hole in the middle