LinkMesh

Search docs, blog and changelog

ENDE

Snippets · Destinations

Send to Kafka (per-signal topics)

Kafka between collectors and backends absorbs bursts and lets several consumers read the same telemetry. The kafka exporter writes each signal to its own topic, OTLP protobuf encoded, so another collector's kafka receiver can read it back unchanged.

Signals
LogsMetricsTraces
Source
OTLP
Components
otlpbatchmemory_limiter
Destination
Apache Kafka

OpenTelemetry Collector

OpenTelemetry Collector YAML
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s

exporters:
  kafka:
    brokers: [kafka-1.example.internal:9092, kafka-2.example.internal:9092]
    logs:
      topic: otlp_logs
      encoding: otlp_proto
    metrics:
      topic: otlp_metrics
      encoding: otlp_proto
    traces:
      topic: otlp_spans
      encoding: otlp_proto
    producer:
      compression: zstd
    sending_queue:
      enabled: true
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [kafka]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [kafka]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [kafka]

LinkMesh

LinkMesh has no built-in template for this job. Add it as a custom destination in raw mode, with kafka as the component type and this body - the same lines as under kafka in the collector config:

Custom destination body (kafka)
brokers: [kafka-1.example.internal:9092, kafka-2.example.internal:9092]
logs:
  topic: otlp_logs
  encoding: otlp_proto
metrics:
  topic: otlp_metrics
  encoding: otlp_proto
traces:
  topic: otlp_spans
  encoding: otlp_proto
producer:
  compression: zstd
sending_queue:
  enabled: true
  queue_size: 5000
retry_on_failure:
  enabled: true
  initial_interval: 5s
  max_interval: 30s
  max_elapsed_time: 300s

How custom components are validated and rolled out: create a custom destination.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0).

The gotcha

Topic and encoding are set per signal now. In our test the older shape with topic and encoding at the top level was refused by otelcol-contrib 0.161.0: "\'kafkaexporter.Config\' has invalid keys: encoding, topic". Use the logs/metrics/traces blocks, as here.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Doing this in LinkMesh: Destinations

Install LinkMesh →See all snippets

Related snippets