Snippets · Destinations

Archive telemetry to S3 (gzip, time-partitioned)

Not all telemetry needs to be searchable today, but some must be kept for a year. The awss3 exporter writes it to object storage as gzipped OTLP JSON, partitioned by time so a later replay or query engine reads only the hours it needs. Credentials come from the standard AWS environment or role, never the file.

OpenTelemetry Collector

OpenTelemetry Collector YAML

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s

exporters:
  awss3:
    s3uploader:
      region: eu-central-2
      s3_bucket: telemetry-archive
      s3_prefix: otel
      # One object per batch under year=/month=/day=/hour=/minute= prefixes.
      s3_partition_format: 'year=%Y/month=%m/day=%d/hour=%H/minute=%M'
      compression: gzip
    marshaler: otlp_json

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [awss3]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [awss3]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [awss3]

LinkMesh

LinkMesh has no built-in template for this job. Add it as a custom destination in raw mode, with awss3 as the component type and this body - the same lines as under awss3 in the collector config:

Custom destination body (awss3)

s3uploader:
  region: eu-central-2
  s3_bucket: telemetry-archive
  s3_prefix: otel
  s3_partition_format: 'year=%Y/month=%m/day=%d/hour=%H/minute=%M'
  compression: gzip
marshaler: otlp_json

How custom components are validated and rolled out: create a custom destination.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0).

The gotcha

Every batch becomes its own object. In our test, against an S3-compatible store, one batch of three log records became one 220-byte object, otel/year=2026/month=10/day=23/hour=06/minute=28/logs_755984280.json.gz. Small batches mean many small objects - size the batch processor up for this pipeline.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Doing this in LinkMesh: Create a custom source or destination

Install LinkMesh →See all snippets

Archive telemetry to S3 (gzip, time-partitioned)

OpenTelemetry Collector

LinkMesh

The gotcha

Manage this across a fleet

Related snippets