Collector High Availability

The OpenTelemetry Collector is easy to stand up and easy to make fragile. A single gateway collector in the path of every signal is a single point of failure: restart it, run it out of memory, or roll a bad config, and telemetry stops — or worse, disappears without anyone noticing until a dashboard is empty during the incident you needed it for. High availability isn’t a feature you toggle on; it’s a set of deliberate choices about redundancy, buffering, and backpressure.

This guide covers those choices for the two tiers most fleets run — agents on every host and a shared gateway pool — and is honest about the guarantee at the end: the Collector gives you at-least-once delivery within finite buffers, not a magic promise that no byte is ever lost.

Two tiers, two HA problems

Most production topologies have two collector tiers, and each fails differently:

• Agent tier — a collector per host or per node (DaemonSet), collecting local telemetry. Its HA problem is local buffering: when the downstream is briefly unreachable or the agent restarts, in-flight data must survive rather than vanish.
• Gateway tier — a pool of collectors that receive OTLP from the agents, do the heavier processing (tail sampling, aggregation, routing), and export to backends. Its HA problem is redundancy and load distribution: no single gateway can be the one whose death stops the pipeline.

The good news is that the mechanisms overlap. Both tiers want a sending queue, retry, and a memory limiter; the gateway tier adds load balancing and stateless replicas on top.

Load-balancing OTLP to a gateway pool

The gateway tier should be a set of identical, stateless replicas behind something that spreads load across them. There are two common approaches, and they solve different problems:

• A plain L4/L7 load balancer or Kubernetes Service. Agents send OTLP to one virtual endpoint; the Service or LB distributes connections across healthy gateway pods. Lose a pod and traffic reroutes to the survivors. This is the right default for metrics and logs, where any gateway can handle any batch.
• The loadbalancing exporter. When you need consistent routing — all spans of a trace landing on the same gateway so tail sampling sees the whole trace — a plain LB isn’t enough, because it may split one trace across pods. The loadbalancing exporter hashes on trace ID (or a routing key) and resolves backends via DNS or a Kubernetes resolver, so related data lands together.

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      k8s:
        service: otel-gateway.observability
        ports: [4317]

Keep gateways stateless so any replica is interchangeable and the pool scales horizontally. State that must be shared (like tail-sampling decisions across a trace) is exactly what the loadbalancing exporter’s routing key keeps on one node, rather than forcing the pool to coordinate.

The sending queue and retry

Redundancy handles a dead node; the sending queue and retry handle a temporarily unreachable one. Every exporter should enable both, so a brief backend outage or a network blip becomes buffering-and-retry instead of dropped data:

exporters:
  otlphttp:
    endpoint: https://backend.example.net
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000

retry_on_failure retries failed exports with exponential backoff up to max_elapsed_time; after that, the batch is dropped rather than retried forever. sending_queue buffers accepted batches in memory so producers aren’t blocked while exports are in flight. Size queue_size for the volume you need to ride out a typical backend hiccup — but remember an in-memory queue is lost on restart, which is the next problem.

A persistent queue that survives restarts

An in-memory queue evaporates when the collector restarts — a deploy, an OOM kill, a node reboot — taking everything buffered with it. The file_storage extension backs the sending queue with disk, so accepted data survives a restart and is retried when the collector comes back:

extensions:
  file_storage/queue:
    directory: /var/lib/otelcol/queue

exporters:
  otlphttp:
    endpoint: https://backend.example.net
    sending_queue:
      enabled: true
      storage: file_storage/queue
      queue_size: 100000

service:
  extensions: [file_storage/queue]

This is what turns “restart drops in-flight telemetry” into “restart pauses delivery briefly.” It matters most on the agent tier, where a node’s local buffer is the only thing standing between a downstream outage and lost data. The cost is disk I/O and a bounded amount of local storage — a trade almost always worth making in the path of data you care about.

Backpressure and the memory limiter

The failure you least want is a collector that accepts more than it can export, grows its heap, and gets OOM-killed — losing its buffers and taking the pipeline down. The memory_limiter processor prevents this by applying backpressure: when memory crosses a soft limit it starts refusing new data, pushing the pressure back to producers (which retry) rather than crashing:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

Put memory_limiter first in the processor chain so it sheds load before work is done on data that can’t be exported anyway. Backpressure feels like refused data, and it is — but a collector that pushes back and stays alive protects far more telemetry than one that swallows everything and dies. Pair it with a graceful drain on shutdown (SIGTERM handling and a termination grace period long enough to flush the queue) so planned restarts flush rather than drop.

What you can and can’t guarantee

Be honest with stakeholders about the guarantee, because overselling it is how trust gets lost during the first real outage:

• You get at-least-once delivery within finite buffers. With retry and a persistent queue, data survives transient failures and restarts and is retried — and may be delivered more than once, so downstream must tolerate duplicates.
• Buffers are bounded, so a long enough outage still drops data. If the backend is down past max_elapsed_time, a batch is given up on; and once the queue fills, newly arriving data is refused and dropped. HA extends how long you can ride out a failure; it doesn’t make the buffer infinite.
• Exactly-once is not on offer. The Collector doesn’t do end-to-end exactly-once delivery. Design dashboards and alerts to tolerate occasional gaps and duplicates rather than assuming a perfect stream.

The right mental model is reduce the blast radius of every likely failure — a node death, a deploy, a spike, a brief backend outage — not guarantee zero loss under all conditions. Get redundancy, queue-plus-retry, persistence, and backpressure in place and you’ve covered the failures that actually happen in production.