Snippets · Pipelines

RED metrics from traces (spanmetrics connector) to Prometheus

Every span already carries what a RED dashboard needs: which service, which operation, whether it failed, how long it took. The spanmetrics connector turns the traces pipeline's spans into two metrics - a calls counter and a duration histogram - per service, span name, kind and status, plus the dimensions you list, and the metrics pipeline remote-writes them to Prometheus. In the CI run below, three spans became two series: two successful POSTs and one failed one.

Included steps

Receive OTLP traces
Forward the traces to the trace backend
Derive calls and duration per series (spanmetrics connector)
Remote-write the metrics

OpenTelemetry Collector

OpenTelemetry Collector YAML

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 1s

connectors:
  spanmetrics:
    # RED metrics per service, span name, kind and status - plus these.
    # Every dimension multiplies the series count: never a user id or full URL.
    dimensions:
      - name: http.request.method
      - name: http.response.status_code
    histogram:
      explicit:
        buckets: [10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s]
    metrics_flush_interval: 15s

exporters:
  otlp/traces:
    endpoint: tempo.example.internal:4317
  prometheusremotewrite:
    endpoint: https://prometheus.example.internal/api/v1/write

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/traces, spanmetrics]
    metrics:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [prometheusremotewrite]

LinkMesh

LinkMesh has no block for connectors today: its pipelines are built from sources, processors and destinations, and a connector is none of the three. Run this config on a collector you configure directly; LinkMesh-managed collectors can still send it the traces or logs it consumes.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0).

Before and after

Example telemetry in, and what this exact config sent out when we ran it in otelcol-contrib v0.161.0. CI re-runs it on every change and fails if the output differs. Changed values are highlighted.

3 spans in, 3 spans out

Before
span
POST /api/orders
attributes.http.request.method
POST
attributes.http.response.status_code
201
attributes.url.full
https://shop.example.ch/api/orders?id=1001
resource.service.name
checkout
After
Dropped
Before
span
POST /api/orders
attributes.http.request.method
POST
attributes.http.response.status_code
201
attributes.url.full
https://shop.example.ch/api/orders?id=1002
resource.service.name
checkout
After
Dropped
Before
span
POST /api/orders
status
ERROR
attributes.http.request.method
POST
attributes.http.response.status_code
503
attributes.url.full
https://shop.example.ch/api/orders?id=1003
resource.service.name
checkout
After
Dropped
Before
(not in the input)
After
span
POST /api/orders
attributes.http.request.method
POST
attributes.http.response.status_code
201
attributes.url.full
https://shop.example.ch/api/orders?id=1001
resource.service.name
checkout
Before
(not in the input)
After
span
POST /api/orders
attributes.http.request.method
POST
attributes.http.response.status_code
201
attributes.url.full
https://shop.example.ch/api/orders?id=1002
resource.service.name
checkout
Before
(not in the input)
After
span
POST /api/orders
status
ERROR
attributes.http.request.method
POST
attributes.http.response.status_code
503
attributes.url.full
https://shop.example.ch/api/orders?id=1003
resource.service.name
checkout

0 metrics in, 4 metrics out (0 data points in, 8 out)

Before
(not in the input)
After
metric
traces.span.metrics.calls
point 1
0 {"service.name":"checkout","span.name":"POST /api/orders","span.kind":"SPAN_KIND_SERVER","status.code":"STATUS_CODE_ERROR","collector.instance.id":"(varies per run)","http.request.method":"POST","http.response.status_code":503}
point 2
0 {"service.name":"checkout","span.name":"POST /api/orders","span.kind":"SPAN_KIND_SERVER","status.code":"STATUS_CODE_UNSET","collector.instance.id":"(varies per run)","http.request.method":"POST","http.response.status_code":201}
resource.service.name
checkout
Before
(not in the input)
After
metric
traces.span.metrics.duration
point 1
{"count":2,"sum":200} {"service.name":"checkout","span.name":"POST /api/orders","span.kind":"SPAN_KIND_SERVER","status.code":"STATUS_CODE_UNSET","collector.instance.id":"(varies per run)","http.request.method":"POST","http.response.status_code":201}
point 2
{"count":1,"sum":1800} {"service.name":"checkout","span.name":"POST /api/orders","span.kind":"SPAN_KIND_SERVER","status.code":"STATUS_CODE_ERROR","collector.instance.id":"(varies per run)","http.request.method":"POST","http.response.status_code":503}
resource.service.name
checkout
Before
(not in the input)
After
metric
traces.span.metrics.calls
point 1
2 {"service.name":"checkout","span.name":"POST /api/orders","span.kind":"SPAN_KIND_SERVER","status.code":"STATUS_CODE_UNSET","collector.instance.id":"(varies per run)","http.request.method":"POST","http.response.status_code":201}
point 2
1 {"service.name":"checkout","span.name":"POST /api/orders","span.kind":"SPAN_KIND_SERVER","status.code":"STATUS_CODE_ERROR","collector.instance.id":"(varies per run)","http.request.method":"POST","http.response.status_code":503}
resource.service.name
checkout
Before
(not in the input)
After
metric
traces.span.metrics.duration
point 1
{"count":2,"sum":200} {"service.name":"checkout","span.name":"POST /api/orders","span.kind":"SPAN_KIND_SERVER","status.code":"STATUS_CODE_UNSET","collector.instance.id":"(varies per run)","http.request.method":"POST","http.response.status_code":201}
point 2
{"count":1,"sum":1800} {"service.name":"checkout","span.name":"POST /api/orders","span.kind":"SPAN_KIND_SERVER","status.code":"STATUS_CODE_ERROR","collector.instance.id":"(varies per run)","http.request.method":"POST","http.response.status_code":503}
resource.service.name
checkout

The gotcha

Every dimension multiplies the series. In our test, adding url.full as a dimension turned the same three spans into three series instead of two - one per order id in the query string; in production that is one series per URL. Keep dimensions to bounded values such as method and status code.

What it does not catch

A new series reports calls 0 on its first flush; in the table, the first flush shows 0 while the histogram already counts the spans, and the next flush (15 s later) shows 2 and 1. That is the counter's starting point, not lost data.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Install LinkMesh →See all snippets