Snippets · Processors

Tail sampling: keep every error and slow trace

Head sampling cannot keep the traces you care about, because it decides before anything has gone wrong. Tail sampling waits for the trace (decision_wait), then applies policies: keep every trace with an error span, every trace slower than 2 seconds, and 5% of the rest. Policies are OR-ed: any match keeps the whole trace.

OpenTelemetry Collector

OpenTelemetry Collector YAML

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # Waits for each trace to complete, then keeps every trace with an error,
  # every trace slower than 2 s, and 5% of the rest.
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch:
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://your-backend:4318

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlphttp]

LinkMesh

LinkMesh has no built-in template for this job. Add it as a custom processor in raw mode, with tail_sampling as the component type and this body - the same lines as under tail_sampling in the collector config:

Custom processor body (tail_sampling)

decision_wait: 10s
num_traces: 50000
policies:
  - name: errors
    type: status_code
    status_code:
      status_codes: [ERROR]
  - name: slow
    type: latency
    latency:
      threshold_ms: 2000
  - name: baseline
    type: probabilistic
    probabilistic:
      sampling_percentage: 5

How custom components are validated and rolled out: create a custom processor.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0). Replace https://your-backend:4318 with your own OTLP endpoint before you run it.

Before and after

Example telemetry in, and what this exact config sent out when we ran it in otelcol-contrib v0.161.0. CI re-runs it on every change and fails if the output differs. Changed values are highlighted.

5 spans in, 3 spans out

Before
span
POST /orders
resource.service.name
checkout
After
span
POST /orders
resource.service.name
checkout
Before
span
charge card
status
ERROR
resource.service.name
payments
After
span
charge card
status
ERROR
resource.service.name
payments
Before
span
GET /search
resource.service.name
checkout
After
span
GET /search
resource.service.name
checkout
Before
span
GET /cart
resource.service.name
checkout
After
Dropped
Before
span
load cart
resource.service.name
payments
After
Dropped

The gotcha

The decision is made per collector, on the spans that collector has seen. All spans of a trace must reach the same collector: in our test a collector that received only the healthy root span of a failing trace dropped it, while one that received the whole trace kept both spans. With more than one collector, route by trace ID first (load-balancing exporter in front of a gateway tier). Expect the decision_wait as added latency before anything is exported.

Used in these pipelines

Agents to a gateway tier, load-balanced by trace ID

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Doing this in LinkMesh: Processors

Install LinkMesh →See all snippets