Five Ways to Manage OpenTelemetry Collectors

TL;DR

• There are five common ways to manage OpenTelemetry Collectors: config management tools, GitOps, the OpenTelemetry Operator, your own OpAMP server with the upstream supervisor, and a dedicated control plane.
• The first three push config and stop there. They know what they sent, not what each collector is actually running.
• OpAMP is the only one that closes the loop: the collector reports its effective config and health back. You either build the server side yourself or run one.
• Kubernetes-only fleets do well with GitOps plus the Operator. Mixed fleets of VMs and clusters are where push-only approaches start to hurt.

You have a few dozen OpenTelemetry Collectors, maybe a few hundred, and the question is no longer how to write a config.yaml. It is how to manage OpenTelemetry Collectors as a fleet: change all of them safely, know what each one is running, and notice when one quietly stops doing its job. Our OpenTelemetry Collector management guide covers what that discipline involves. This post compares the tools people actually use for it.

Five approaches come up again and again. They are not mutually exclusive, and plenty of teams run two of them side by side. What separates them is one question: does information flow back from the collector, or only to it?

Figure 1. Three approaches push config and stop. Two close the loop over OpAMP, so the collector reports back what it is running.

What managing OpenTelemetry Collectors has to cover

Before comparing tools, it helps to name the jobs. Whatever you pick has to answer these, or leave them to a human:

• Deliver config. Get the right pipeline onto the right collector, and restart or reload it so the change takes effect.
• Know what is running. Not what you sent: what the collector actually loaded, which version it is, and whether it is healthy.
• Catch drift. Someone edits a file on a host during an incident. Something has to notice and either correct it or tell you.
• Roll out safely. A bad processor config can drop every log in a region. You want to change a few collectors first, and undo quickly.
• Handle a mixed fleet. Most real fleets are VMs, bare metal and one or more Kubernetes clusters at the same time.

Keep those five jobs in mind. Each approach below handles some of them well and leaves the rest to you.

1. Config management: Ansible, Puppet, Chef, Salt

This is where most teams start, because the tooling is already there. You template the collector config, push it to hosts, and a handler restarts the service when the file changes.

What it does well. It works on any host you can reach, it fits the change process you already have, and a pull request on the playbook is a reasonable review step. For a few dozen VMs with a handful of config variants, it is often enough.

Where it stops. Config management is fire and forget. It knows what it wrote to disk on the last run, not what the collector loaded or whether the pipeline is moving data. A broken exporter config restarts cleanly and then drops data, and the playbook reports success. Drift is only corrected on the next run, and only for the files the playbook owns. Kubernetes collectors sit outside it entirely.

2. GitOps: Argo CD or Flux on Kubernetes

On Kubernetes, the natural move is to keep the collector config in Git as a ConfigMap or Helm values and let Argo CD or Flux reconcile the cluster to it. We wrote up the pattern in detail in GitOps for OpenTelemetry Collector config.

What it does well. Git becomes the single history of every change, review happens in pull requests, and the reconciler corrects drift in the manifests continuously. Rollback is a revert.

Where it stops. The reconciler watches Kubernetes objects, not collectors. “Synced” means the ConfigMap matches Git. It does not mean every pod loaded it. A changed ConfigMap does not restart the pods by itself, so teams add a checksum annotation to force a rollout. And GitOps covers only the clusters your reconciler runs in. The VMs at the edge still need a second approach.

3. The OpenTelemetry Operator

The OpenTelemetry Operator adds an OpenTelemetryCollector custom resource. You describe the collector and its config, and the Operator creates the Deployment, DaemonSet, StatefulSet or sidecar that runs it. It pairs naturally with GitOps: Git holds the custom resources, the reconciler applies them, the Operator turns them into running collectors.

What it does well. It removes a lot of Kubernetes boilerplate, handles sidecar injection, and gives you one declarative object per collector deployment. The project also ships an OpAMP Bridge, which can report the collectors the Operator manages to an OpAMP server.

Where it stops. It is Kubernetes only, by design. On its own it is still a push model. You learn what the collector is really doing through your normal monitoring, not from the Operator. For a fleet that is entirely in Kubernetes, GitOps plus the Operator is a strong combination. For anything else it is one half of the answer.

4. Your own OpAMP server with the supervisor

OpAMP, the Open Agent Management Protocol, is the OpenTelemetry project’s answer to the missing feedback loop. Each collector holds a connection to a management server, reports its identity, its effective config and its health, and can receive new config over the same connection.

The collector-contrib repository ships an OpAMP supervisor. It runs next to the collector, accepts remote config from the server, writes it, and restarts the collector. The protocol libraries, for example opamp-go, give you the client and server building blocks. The setup is walked through in OpAMP supervisor step by step.

What it does well. This is the only approach on the list where the collector itself tells you what it is running. It works the same on a VM and in a pod, so one mechanism covers a mixed fleet. And it is vendor-neutral: you are building on an OpenTelemetry protocol, not a product.

Where it stops. The supervisor and the protocol are the client side. The server is yours to build: storing configs, deciding which collector gets which, tracking versions, showing state, handling enrollment and credentials, and deciding what happens when a config is rejected. The supervisor is also still marked alpha in collector-contrib at the time of writing, and some OpAMP features, such as package management, are not implemented in it yet. This is a good choice for a team that wants to own a platform. It is a real engineering project, not a weekend.

5. A collector control plane

The fifth approach is to run an OpAMP server someone else has built. A collector control plane is that server plus the parts around it: an inventory of every collector, config storage and history, a way to push to one collector or a group, health and throughput views, and enrollment so a new collector can join the fleet with a credential.

What it does well. You get the closed loop of approach 4 without writing the server. Mixed fleets show up in one view, and drift becomes something the system notices rather than something you discover during an incident.

Where it stops. You are now running, or paying for, one more component, and it sits in the path of every config change. Where it runs matters as much as what it does. A control plane hosted by a vendor holds your fleet inventory and your collector credentials outside your network. That is fine for some teams and ruled out for others. Products also differ in what they offer for staged rollouts and collector upgrades, so check those against your own requirements rather than assuming them.

Side by side: which approach fits

Figure 2. Only the OpAMP approaches report what each collector actually runs. The difference between them is who builds the server.

Three questions decide most cases:

1. Is the whole fleet in Kubernetes? If yes, GitOps plus the Operator covers config delivery, review and drift well. Add monitoring of the collectors’ own telemetry and you may not need more.
2. Do you need to know what each collector is actually running? If the answer matters for compliance or for incident response, you need OpAMP in some form. Push-only approaches cannot tell you.
3. Do you want to build the server? If you have a platform team that wants to own it, the supervisor and the OpAMP libraries are a solid foundation. If not, run a control plane, and decide where it is allowed to run before you compare features.

Most teams with a mixed fleet end up combining approaches: config management or GitOps to install the collector and the supervisor, and OpAMP to manage the config from then on. That split works well, because each tool does the job it is good at. Our post on collector deployment patterns covers the agent and gateway layouts this sits on.

Where LinkMesh fits

LinkMesh is a self-hosted collector control plane, approach 5. It manages upstream otelcol-contrib collectors through the upstream OpAMP supervisor, so there is no proprietary agent in the config path. VMs and Kubernetes clusters show up in the same inventory, grouped the way you run them.

The control plane notices when a collector reports a config that differs from the one you committed, flags it on the health page, and pushes the committed config again. It runs as a single binary with embedded storage, on your own infrastructure, so the fleet inventory and collector credentials stay inside your network.

If you are weighing approach 4 against approach 5, the OpAMP server page shows what the server side covers. To try it on your own collectors, install LinkMesh and enroll one collector to watch the loop close.