Snippets · Processors

Sample traces probabilistically (head sampling)

Tracing everything is expensive and most traces look alike. The probabilistic sampler keeps a percentage of traces, decided by a hash of the trace ID, so all spans of a trace get the same decision on every collector without coordination. It cannot know whether a trace failed; for that, see tail-sampling-errors.

OpenTelemetry Collector

OpenTelemetry Collector YAML

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # Keeps about 10% of traces. The decision is a hash of the trace ID, so
  # every collector keeps or drops the same traces, whole.
  probabilistic_sampler:
    sampling_percentage: 10
  batch:
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://your-backend:4318

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, probabilistic_sampler, batch]
      exporters: [otlphttp]

LinkMesh

Library › Processors › Probabilistic Trace Sampling

sampling_percentage: 10

What LinkMesh then pushes to the collector, under processors:

Pushed to the collector

probabilistic_sampler/route_00000506_pipe_proc_1:
  sampling_percentage: 10

The same, as an API call (see automation recipes):

API call

curl -X POST "$LINKMESH/pipelines" \
  -H "Authorization: Bearer $LMSAT" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "Probabilistic Trace Sampling",
  "processors": [
    {
      "config": {},
      "enabled": true,
      "order": 1,
      "templateSlug": "probabilistic-sampling"
    }
  ]
}'

The rest of the collector config above (otlp, memory_limiter, batch, otlphttp) is not part of this template: in LinkMesh those are separate sources, processors and destinations on the route.

Generated from LinkMesh's template catalogue by linkmesh-server snippet-render 1.464.3, not written by hand.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0). Replace https://your-backend:4318 with your own OTLP endpoint before you run it.

Before and after

Example telemetry in, and what this exact config sent out when we ran it in otelcol-contrib v0.161.0. CI re-runs it on every change and fails if the output differs. Changed values are highlighted.

20 spans in, 1 span out

Before
span
GET /orders/1000
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1001
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1002
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1003
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1004
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1005
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1006
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1007
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1008
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1009
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1010
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1011
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1012
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1013
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1014
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1015
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1016
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1017
resource.service.name
checkout
After
span
GET /orders/1017
resource.service.name
checkout
Before
span
GET /orders/1018
resource.service.name
checkout
After
Dropped
Before
span
GET /orders/1019
resource.service.name
checkout
After
Dropped

The gotcha

The percentage holds on volume, not on a small sample: our 20 test traces kept 1. And the same processor in a logs pipeline samples by trace ID too, so logs without one have nothing to hash: in our test it kept 0 of 20 log records that had no trace ID.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Doing this in LinkMesh: Processors

Install LinkMesh →See all snippets