Snippets · Pipelines

Agents to a gateway tier, load-balanced by trace ID

Tail sampling only works when one collector sees every span of a trace. Agents next to the applications use the loadbalancing exporter with routing_key traceID: it resolves all gateway instances through a headless Service and sends each trace to one of them, where the gateway config (the second file) runs tail sampling. We ran one agent and two gateways: 8 traces of 3 spans each were spread over both gateways (5 and 3 traces), and not one trace was split between them.

Included steps

Agents receive OTLP from applications
Route by trace ID to the gateway tier (loadbalancing exporter)
Gateways tail-sample complete traces
memory_limiter first, batch last

OpenTelemetry Collector

Agent collector YAML

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 1s

exporters:
  # Sends all spans of one trace to the same gateway, so tail sampling there
  # sees whole traces. The DNS name resolves to every gateway instance.
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-gateway-headless.observability.svc.cluster.local
        port: "4317"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loadbalancing]

Gateway collector YAML

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # Waits for each trace to complete, then keeps every trace with an error,
  # every trace slower than 2 s, and 5% of the rest.
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch:
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://your-backend:4318

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlphttp]

LinkMesh

LinkMesh has no built-in template for this job. Add it as a custom destination in raw mode, with loadbalancing as the component type and this body - the same lines as under loadbalancing in the collector config:

Custom destination body (loadbalancing)

routing_key: traceID
protocol:
  otlp:
    tls:
      insecure: true
resolver:
  dns:
    hostname: otel-gateway-headless.observability.svc.cluster.local
    port: "4317"

How custom components are validated and rolled out: create a custom destination.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0). Replace https://your-backend:4318 with your own OTLP endpoint before you run it.

The gotcha

The resolver port is a string. In our test port: 4317 made the agent config fail to load with "'port' expected type 'string', got unconvertible type 'int'" - it has to be "4317" in quotes. The gateway side is the tail-sampling snippet; why whole traces matter is measured there.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Install LinkMesh →See all snippets