Snippets · Pipelines

Buffer through a backend outage with a persistent queue

Backends go down; telemetry keeps arriving. With sending_queue.storage pointing at a file_storage extension, batches that cannot be delivered wait on disk instead of in memory, and retry_on_failure with max_elapsed_time: 0 keeps retrying for as long as the outage lasts. CI replays exactly that on every change: the collector starts with no backend listening, is restarted half-way through the outage, and only then does the backend start - the after column below is what the backend received.

Included steps

OpenTelemetry Collector

OpenTelemetry Collector YAML

extensions:
  file_storage/queue:
    directory: /var/lib/otelcol/queue
    create_directory: true

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 1s

exporters:
  otlp/backend:
    endpoint: backend.example.internal:4317
    tls:
      insecure: false
    sending_queue:
      enabled: true
      # Batches waiting for the backend live on disk, so they survive a
      # collector restart during the outage.
      storage: file_storage/queue
      queue_size: 10000
    retry_on_failure:
      enabled: true
      initial_interval: 2s
      max_interval: 10s
      # 0 keeps retrying for as long as the outage lasts.
      max_elapsed_time: 0s

service:
  extensions: [file_storage/queue]
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]

LinkMesh

Library › Destinations › OTLP gRPC

Endpoint: backend.example.internal:4317
Compression: gzip
Timeout: 30s
Durable Delivery: true

What LinkMesh then pushes to the collector, under exporters:

Pushed to the collector

otlp/output_dst_00000502:
  compression: gzip
  endpoint: backend.example.internal:4317
  sending_queue:
    block_on_overflow: false
    enabled: true
    storage: file_storage/output_dst_00000502
  timeout: 30s

# extensions:
file_storage/output_dst_00000502:
  create_directory: true
  directory: /var/lib/otelcol-supervisor/sending-queue/output_dst_00000502

The same, as an API call (see automation recipes):

API call

curl -X POST "$LINKMESH/destinations" \
  -H "Authorization: Bearer $LMSAT" \
  -H "Content-Type: application/json" \
  -d '{
  "defaultConfig": {
    "compression": "gzip",
    "durable_queue": true,
    "endpoint": "backend.example.internal:4317",
    "timeout": "30s"
  },
  "exporterType": "otlp",
  "name": "OTLP gRPC",
  "signalTypes": [
    "logs",
    "metrics",
    "traces"
  ]
}'

The rest of the collector config above (memory_limiter, batch) is not part of this template: in LinkMesh those are separate sources, processors and destinations on the route.

Generated from LinkMesh's template catalogue by linkmesh-server snippet-render 1.464.3, not written by hand.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0).

Before and after

Example telemetry in, and what this exact config sent out when we ran it in otelcol-contrib v0.161.0. CI re-runs it on every change and fails if the output differs. Changed values are highlighted.

3 log records in, 3 log records out

Before
body
order 1001 accepted during the outage
timestamp
2025-10-09T08:53:20.000Z
severity
INFO
severity_number
9
resource.service.name
checkout
After
body
order 1001 accepted during the outage
timestamp
2025-10-09T08:53:20.000Z
severity
INFO
severity_number
9
resource.service.name
checkout
Before
body
order 1002 accepted during the outage
timestamp
2025-10-09T08:53:21.000Z
severity
INFO
severity_number
9
resource.service.name
checkout
After
body
order 1002 accepted during the outage
timestamp
2025-10-09T08:53:21.000Z
severity
INFO
severity_number
9
resource.service.name
checkout
Before
body
payment provider timeout during the outage
timestamp
2025-10-09T08:53:22.000Z
severity
ERROR
severity_number
17
resource.service.name
checkout
After
body
payment provider timeout during the outage
timestamp
2025-10-09T08:53:22.000Z
severity
ERROR
severity_number
17
resource.service.name
checkout

The gotcha

Without storage the queue lives in memory and a restart during the outage loses it. In the same replay with only the storage line removed, the backend received 0 of the 3 records; with it, all 3 arrived after the restart. Give file_storage a directory that survives the restart (a volume, not the container's filesystem).

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Install LinkMesh →See all snippets