Snippets · Processors

Deduplicate repeated log records

A failing dependency produces the same error line hundreds of times a minute. The logdedup processor collapses identical records seen within an interval into one, adds log_count with the number of copies and the first and last observed time, and sends it at the end of the interval.

OpenTelemetry Collector

OpenTelemetry Collector YAML

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # Collapses identical log records seen within 10 s into one, with the number
  # of copies in log_count. request.id differs on every copy, so it is left
  # out of the comparison.
  logdedup:
    interval: 10s
    log_count_attribute: log_count
    exclude_fields:
      - attributes.request\.id
  batch:
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://your-backend:4318

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, logdedup, batch]
      exporters: [otlphttp]

LinkMesh

LinkMesh has no built-in template for this job. Add it as a custom processor in raw mode, with logdedup as the component type and this body - the same lines as under logdedup in the collector config:

Custom processor body (logdedup)

interval: 10s
log_count_attribute: log_count
exclude_fields:
  - attributes.request\.id

How custom components are validated and rolled out: create a custom processor.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0). Replace https://your-backend:4318 with your own OTLP endpoint before you run it.

Before and after

Example telemetry in, and what this exact config sent out when we ran it in otelcol-contrib v0.161.0. CI re-runs it on every change and fails if the output differs. Changed values are highlighted.

6 log records in, 2 log records out

Before
body
upstream connect error: connection refused
severity
ERROR
severity_number
17
attributes.server.address
payments
attributes.request.id
r-0
resource.service.name
checkout
After
body
upstream connect error: connection refused
timestamp
(varies per run)
severity
ERROR
severity_number
17
attributes.server.address
payments
attributes.log_count
5
attributes.first_observed_timestamp
(varies per run)
attributes.last_observed_timestamp
(varies per run)
resource.service.name
checkout
Before
body
upstream connect error: connection refused
severity
ERROR
severity_number
17
attributes.server.address
payments
attributes.request.id
r-1
resource.service.name
checkout
After
Dropped
Before
body
upstream connect error: connection refused
severity
ERROR
severity_number
17
attributes.server.address
payments
attributes.request.id
r-2
resource.service.name
checkout
After
Dropped
Before
body
upstream connect error: connection refused
severity
ERROR
severity_number
17
attributes.server.address
payments
attributes.request.id
r-3
resource.service.name
checkout
After
Dropped
Before
body
upstream connect error: connection refused
severity
ERROR
severity_number
17
attributes.server.address
payments
attributes.request.id
r-4
resource.service.name
checkout
After
Dropped
Before
body
order 1002 accepted
severity
INFO
severity_number
9
attributes.request.id
r-9
resource.service.name
checkout
After
body
order 1002 accepted
timestamp
(varies per run)
severity
INFO
severity_number
9
attributes.log_count
1
attributes.first_observed_timestamp
(varies per run)
attributes.last_observed_timestamp
(varies per run)
resource.service.name
checkout

The gotcha

A field with a dot in its name must be escaped in exclude_fields: attributes.request\.id, not attributes.request.id. In our test the unescaped form matched nothing, without an error: 6 records in, 6 out. Escaped, the five copies collapsed: 6 in, 2 out, log_count 5. Excluded fields are removed from the output, and records arrive up to one interval late.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Doing this in LinkMesh: Create a custom processor

Install LinkMesh →See all snippets