Elastic Agent → OpenTelemetry

The Elastic Stack is a capable, widely deployed observability and logging platform, and for a lot of teams it earned its place. Two things push teams to look at the collection layer specifically: the cost that scales with ingest and storage as data volume grows, and the agent estate — a fleet of Beats and Fleet-managed Elastic Agents that speak Elastic’s protocol and, for the most part, feed Elasticsearch and nothing else.

The good news is that modernizing collection no longer means a rip-and-replace. With OpenTelemetry you can put a vendor-neutral collection layer under your existing pipeline, keep Elasticsearch and Kibana running as one destination among several, and cut the volume you pay to index — all before you commit to anything. Elastic itself now ships its own collector build under the Elastic Distributions of OpenTelemetry (EDOT) umbrella and Elasticsearch has an OTLP-capable ingest path, so this is a supported direction, not a hack. The broader case is in Reducing vendor lock-in with OpenTelemetry; this post is the Elastic-specific playbook.

First: which “Elastic” are we migrating?

“Elastic to OpenTelemetry” is ambiguous, and getting specific up front saves you a scoping mistake. This guide is about the collection layer — the agents that get data into Elasticsearch. Know which components you actually run:

• Beats — the single-purpose shippers: Filebeat (logs), Metricbeat (metrics), Packetbeat, Auditbeat, Winlogbeat, Heartbeat. These are the biggest thing OTel replaces.
• Elastic Agent — the unified agent that bundles the Beats behind one binary and one policy, replacing separate Beats installs.
• Fleet — the Kibana-based control plane that manages Elastic Agents centrally (policies, enrollment, upgrades). Its role maps to an OpAMP control plane.
• Ingest pipelines — Elasticsearch-side processors (grok, dissect, enrich). Some of this parsing moves to Collector processors; some stays index-side.
• EDOT Collector — Elastic’s own supported OTel Collector build, part of the Elastic Distributions of OpenTelemetry (EDOT) family alongside EDOT SDKs. If you already run it, you’re partly on OTel already; this guide’s processor and routing work still applies, you just choose the exporter.

One disambiguation: this targets self-managed or Elastic Cloud ingestion into Elasticsearch/Kibana. If you want the bigger-picture argument for why this is possible at all, start with what OpenTelemetry can and cannot replace.

What migrates to OpenTelemetry — and what doesn’t

OTel replaces the collection and forwarding layer. It does not replace Elasticsearch’s index, Kibana’s UI, or ES|QL — those stay in Elastic if you keep it, or get rebuilt if you leave:

Elastic capability	Move to OTel?	Notes
File / log collection (Filebeat)	Yes	filelog receiver replaces Filebeat.
Host / system metrics (Metricbeat)	Yes	hostmetrics (+ kubeletstats) replace Metricbeat’s system modules.
Windows event logs (Winlogbeat)	Yes	windowseventlog receiver.
Syslog ingest	Yes	syslog receiver (RFC 3164 / 5424).
APM (Elastic APM agents)	Usually	OTel SDKs / auto-instrumentation emit OTLP; validate span and service naming.
Ingest-pipeline parsing (grok, dissect)	Often	Recreate as transform / filter processors (OTTL); some stays index-side.
Fleet fleet-management	Yes (as OpAMP)	Central agent management maps to an OpAMP control plane, not to Elastic Fleet.
Indexing & storage	No	Elasticsearch, or a replacement backend. Not an OTel concern.
ES / ES|QL / KQL queries	No	Query language is backend-specific; rebuild if you leave Elastic.
Kibana dashboards	No	Rebuild in the new backend, or keep Kibana for them.
Watcher / Kibana alerting	No	Migrate separately to the new backend’s alerting.

The rule of thumb: OTel handles collection (logs, metrics, traces) and fleet management (via OpAMP) cleanly. Elastic-specific products — ES|QL, Kibana dashboards, Watcher — are not things you “migrate”; you keep them, replace them, or retire them.

Map the Elastic pieces to their OTel equivalents

Most of the migration is a one-to-one substitution. Every Beat and Elastic Agent input has an OpenTelemetry counterpart:

Elastic	OpenTelemetry equivalent
Filebeat (files, containers)	OTel Collector with the filelog receiver
Metricbeat (system, modules)	hostmetrics + kubeletstats + prometheus receivers
Winlogbeat (WinEventLog)	windowseventlog receiver
Syslog input	syslog receiver (RFC 3164 / 5424)
Elastic APM agents	OTel SDKs / auto-instrumentation emitting OTLP
Ingest pipelines (grok, dissect, enrich)	transform / filter / attributes processors (OTTL)
Fleet (central agent management)	OpAMP control plane
Elasticsearch (destination)	elasticsearch exporter — or any OTLP backend

Note the last two rows — they are the escape hatches. The Collector’s elasticsearch exporter writes straight into your existing Elasticsearch cluster, so you do not have to move off the Elastic Stack to adopt OpenTelemetry — you put OTel in front of it first and decide about the backend later. And Fleet’s central management maps to OpAMP, so you don’t lose fleet control when you drop the Elastic Agent; you move it to an open standard. That decoupling is the whole trick.

Here’s what that looks like in practice — collector sources configured centrally instead of one Beat config per host:

Step 1 — Stand up collectors next to the Beats

Don’t touch the Beats or Elastic Agents yet. Deploy OTel Collectors alongside them, reading the same sources, and export to Elasticsearch via the elasticsearch exporter. A minimal filelog-to-Elasticsearch collector looks like this:

receivers:
  filelog:
    include: [/var/log/*.log]
    start_at: end
  hostmetrics:
    collection_interval: 30s
    scrapers: { cpu: , memory: , disk: , network: , load: }

exporters:
  elasticsearch:
    endpoints: ["https://elasticsearch.internal:9200"]
    logs_index: "otel-logs"

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [elasticsearch]
    metrics:
      receivers: [hostmetrics]
      exporters: [elasticsearch]

(One note if you manage this through LinkMesh: its built-in Elasticsearch destination carries logs and traces; host metrics are typically routed to a metrics backend like Prometheus or Grafana Cloud instead.)

At this point nothing has changed for your users — data still lands in Elasticsearch and Kibana still queries it — but the collection layer is now open. Every improvement from here is one you couldn’t make from inside a Beat.

Step 2 — Dual-ship and compare

Add a second exporter. Keep writing to Elasticsearch, and start writing the same stream to a candidate OTLP backend (Grafana Loki, an OTLP-native store, or object storage). Now you can compare them side by side on real production data — searches, dashboards, alert coverage — with zero risk, because Elasticsearch is still the system of record.

exporters:
  elasticsearch:
    endpoints: ["https://elasticsearch.internal:9200"]
    logs_index: "otel-logs"
  otlphttp/candidate:
    endpoint: "https://backend.internal:4318"

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [elasticsearch, otlphttp/candidate]

This is the move a proprietary Beat simply can’t make: send one stream to two backends at once. It’s what turns an Elastic migration into a gradual, reversible process instead of a flag day.

Step 3 — Cut the volume you were paying to index

This is where the migration pays for itself. Elastic cost scales with ingest and storage, and a large fraction of what you index is noise you never search. In a Beat you could do a little with processors; in an OTel Collector you can do a lot, in a place you control.

Drop sub-INFO logs before they ever reach an index:

processors:
  filter/drop_debug:
    logs:
      log_record:
        - 'severity_number < SEVERITY_NUMBER_INFO'

Sample a high-volume, low-value stream down to a representative fraction:

processors:
  probabilistic_sampler:
    sampling_percentage: 20

Every record dropped at the collector is a record you don’t pay to ingest, index, and retain — in Elasticsearch or any per-GB backend. This is the single biggest lever in the whole migration — see reducing observability costs for the full filter/sample/route playbook. You can also mask sensitive fields before they leave the host — card numbers, emails, tokens — which shrinks payloads and keeps PII out of the backend at the same time, covered in PII masking in logs.

Step 4 — Manage the fleet with OpAMP, not Fleet

Here’s the catch worth naming: the Elastic Agent’s whole value proposition is Fleet — central policies, enrollment, upgrades from Kibana. Drop the Elastic Agent for raw OTel Collectors and you lose that, hand-syncing YAML across every node — a step backwards in manageability.

Don’t accept that trade. Fleet’s job maps directly to OpAMP, the open protocol for managing an agent fleet. LinkMesh is a self-hosted OpAMP control plane: point each collector at it with a token, then build sources, processors, and routes in a visual UI. It renders and validates the config and pushes it to the right nodes — the Fleet experience, but for open-standard OTel Collectors, and with per-edge throughput so you can see the volume you just cut. Because it’s priced per collector, not per GB, the tool managing your escape from volume pricing isn’t itself metered by volume. Every config change is versioned and auditable, GitOps-style — see GitOps for collector config.

Validate parity before you cut over

Dual-shipping only de-risks the migration if you actually check the two streams match. Before removing anything, work a validation checklist:

• Volume — compare document/event counts per source between Elasticsearch and the candidate over the same window. A large gap means a dropped source or an over-aggressive filter.
• Completeness — confirm every input a Beat or Elastic Agent collected has a matching receiver, including Windows event logs and any custom modules.
• Parsing — spot-check that timestamps, severities, and key extracted fields land correctly (OTTL parsing differs from grok/dissect ingest pipelines; this is the most common gap).
• Attribution — verify service.name, host, and environment resource attributes are present so data is queryable the way your dashboards and searches expect.
• Sampling — confirm sampled streams keep 100% of errors and security events; only the high-volume, low-value majority should thin out.
• Dropped telemetry — check the Collector’s own internal metrics for refused or dropped records; a silent exporter failure looks like “less noise” until an incident.
• Downstream inputs — re-point (or rebuild) the Kibana dashboards and alerts that matter most against the candidate and confirm they populate and fire.
• Failure behavior — restart a collector and kill a backend connection; confirm the Collector buffers and resumes without losing data or wedging.

Have a rollback plan

Because you dual-shipped, rollback is built in — but make it explicit before you flip anything:

• Keep the Beats / Elastic Agents running (or easily re-enablable) through the whole validation window. Don’t uninstall on day one.
• Route independently. The Elasticsearch exporter and the candidate exporter are separate blocks; removing one never touches the other, so you can revert a single stream.
• Define success thresholds up front — e.g. “volume within 2%, all priority alerts firing, no parsing regressions for 7 days” — so cutover is a decision, not a vibe.
• Retain the ability to redirect. If the candidate misbehaves, add the elasticsearch exporter back to the pipeline and you’re fully on Elastic again in one config push.
• Decommission only after an agreed observation period — typically one to two full business cycles after cutover, not the moment parity first looks right.

Step 5 — Cut over on your own schedule

When the candidate backend has proven itself — dashboards match, alerts fire, the searches your team relies on work — flip the exporter. Remove the elasticsearch exporter (or keep it for a subset of data you genuinely want in Elastic), and decommission the Beats and Elastic Agents stream by stream. Because instrumentation and collection are now OTLP and OTel, there’s nothing to re-instrument; it’s an exporter change.

And the migration is genuinely done, not half-done: your telemetry now speaks an open protocol, your agents are managed on an open standard (OpAMP, not Fleet), and the next backend decision — if there ever is one — is another exporter block, not another year-long project.

The migration in five moves

1. Substitute — deploy OTel Collectors next to the Beats, exporting to Elasticsearch via the elasticsearch exporter. Nothing changes for users.
2. Dual-ship — add a candidate OTLP backend and compare on live data.
3. Refine — filter, sample, and mask at the collector to cut ingest volume.
4. Validate & keep a rollback — check parity against the checklist above, with the Beats still in place as your safety net.
5. Cut over — flip the exporter when parity holds; retire the Beats and Elastic Agents stream by stream after an observation period, with fleet management now on OpAMP.

You can run all five with raw OTel Collectors and a lot of YAML, or you can manage the fleet from a self-hosted OpAMP control plane and skip the hand-syncing. Either way, the collection-layer lock-in is gone the moment your data speaks OTLP.