Detecting Config Drift

You believe your OpenTelemetry Collector fleet runs a known set of policies — PII masked, noise filtered, attributes standardized, data routed only to approved backends. Drift is the gap between that belief and reality: one or more nodes whose running config has quietly diverged from the config you intended. Nothing is on fire, no alert fires — the node is happily running a config that violates a policy you thought was universal.

That silence is what makes drift dangerous. A crashed collector is obvious; a collector that’s been running an unredacted pipeline since last Tuesday’s incident is invisible until an auditor, a bill, or a breach finds it for you. This guide covers how drift happens, why it’s costly, how to detect it, and how to stop it happening at all.

How drift happens

Drift is rarely malicious. It’s the accumulated residue of normal operational life:

• Hand-edits during incidents. At 3 a.m. someone SSHes into a node, tweaks a filter to stop a flood or bumps a queue size, restores service — and never reverts. The emergency edit becomes permanent by inattention.
• Partial rollouts. A config push fails on a subset of nodes, or a canary was never expanded, so part of the fleet runs the new version and part runs the old one. Both are “valid”; neither is uniform.
• Manual hotfixes never reverted. A one-off change to unblock a team gets forgotten once the pressure is off. The fix outlives the reason for it.
• Version skew. New nodes come up from a template that’s a few versions behind, or an autoscaler launches instances with a stale baked-in config. The fleet slowly stratifies into config generations.

None of these throws an error. Each just leaves a node running something other than what you think it runs.

Why drift is dangerous

The reason drift matters is that the things a collector config controls are exactly the things you can’t afford to get silently wrong:

• Silent policy violations. A node whose redaction processor was edited out is shipping unredacted PII across the network — a compliance incident that produces no error and no alert, only a liability.
• Cost blowouts. A filter or sampler that drifted off — or was loosened during an incident — lets a high-volume stream through unsampled, and the first sign is next month’s bill, undoing the work in reducing observability costs.
• Unattributed data. A node missing its resource attributes emits telemetry with no service.name, environment, or owner, so it can’t be attributed, queried, or charged back — and quietly pollutes dashboards for everyone.

The common thread is that drift breaks governance without breaking anything visible. The policy is still true on paper and on most of the fleet; it’s just not true here, and nothing tells you.

How to detect drift

Detecting drift means continuously answering one question: does the config each node is actually running match the config we intended for it? A few complementary techniques:

• Compare running config against the source of truth. Pull each node’s effective config and diff it against the versioned config it’s supposed to run. Any difference is drift, full stop.
• Version and hash checks. Attach a version or a content hash to every config and have each node report the hash it’s running. Comparing reported hash to intended hash is a cheap, continuous drift check across the whole fleet — a mismatch is a flag without a full diff.
• Config-apply audit events. Record every apply — who, what, when, which version — so an out-of-band change (a hand-edit that didn’t come through the normal path) shows up as a node whose running state has no corresponding apply event.

# Hash the config a node is actually running and compare to intended
sha256sum /etc/otelcol/config.yaml | awk '{print $1}'
# -> 9f7c...   (intended: a1b2...)  => DRIFT

Ad-hoc scripts like this work for a handful of nodes, but they’re a snapshot, not a guard — they tell you about drift after it happened and only when you remember to run them. The stronger position is a fleet where drift can’t take hold in the first place.

How a control plane prevents drift

Detection tells you drift happened; prevention stops it happening. The structural fix is to make the central config authoritative so that a node’s local state can’t diverge and stay divergent. This is precisely what OpAMP, the Open Agent Management Protocol, enables: a central server owns each collector’s config and pushes it to the fleet. When the server is the source of truth, local hand-edits stop being authoritative — the assigned config is what runs, and an out-of-band edit is either overwritten on the next sync or surfaced as a mismatch rather than silently persisting.

That inverts the whole problem. With raw YAML files, the running config is authoritative and the repo is a hopeful copy; drift is the default and detection is a chore. With a control plane, the central version is authoritative and each node’s job is to match it; drift is the exception and it’s visible by construction.

LinkMesh is a self-hosted control plane built on exactly this model. The config you compose centrally is the source of truth; it’s rendered, validated, and pushed over OpAMP to the right nodes, and because the control plane owns it, a hand-edit on a box isn’t the authority — the enforced config is. Every change is versioned and audited (GitOps style), so “which nodes match intended, and who changed what, when” always has an answer. It’s the same mechanism behind safe rollout and rollback, GitOps for collector config, and governance enforcement: one authoritative config, applied uniformly, drift-free by design rather than by vigilance.

Where to start

You don’t need a control plane to begin — you need to stop being surprised:

1. Establish an intended config. Version it. If there’s no source of truth, there’s nothing to drift from and nothing to detect against.

2. Make each node report what it runs. A version or hash per node turns “are we in sync?” into a comparison instead of a guess.

3. Compare continuously and alert on mismatch. A daily diff of running-vs-intended catches drift within a day instead of at audit time.

   Drift that changes what a collector carries also shows up in its throughput: a filter that drifted on quietly drops volume, one that drifted off quietly adds it. A threshold on records per second catches that class of drift the moment it changes the data, without waiting for the next diff.

   

4. Then make central config authoritative. Move to OpAMP-managed config so local edits can’t be the authority — which converts drift from something you detect into something that mostly can’t happen.

Drift is the slow leak in every hand-managed fleet. You can bail it out with detection, or you can seal it with an authoritative control plane — and the second is a lot less work at 3 a.m.