GitOps for Observability

Platform teams learned years ago not to hand-edit production by SSH. Infrastructure became code: versioned, reviewed, rolled back. Observability config somehow didn’t get the memo — plenty of fleets still run on YAML files edited in place, host by host, with no history and no way to know what’s actually deployed where.

This post is about applying GitOps to observability: what it means to treat collector configuration as code, why a git repo alone only half-solves the problem, and what a git-backed control plane adds — commit/publish workflows, version history, and the drift detection that plain GitOps can’t give you.

The config-drift problem

Picture 30 hosts, each running an OpenTelemetry Collector, each with its own YAML file. Someone tweaks a processor on host 12 during an incident and forgets to write it down. A playbook run half-applies because three hosts were unreachable. A new hire copies an old config to a fresh node. Within weeks, “the fleet’s configuration” isn’t one thing — it’s 30 slightly different things, and nobody can say with confidence what any given host is running.

That’s config drift, and it’s corrosive: it makes incidents harder to reason about (“is this host even on the current config?”), it makes changes risky, and it quietly undoes the standardization you were trying to enforce.

Why plain GitOps only half-solves it

The obvious fix is a git repo of collector configs plus a CI job that pushes them out. That’s a real improvement — you get history and review on the files. But a repo of YAML has three gaps that matter for observability specifically:

• No validation. Git will happily version a config with a typo’d receiver or an invalid pipeline. You find out when a collector fails to start and ingestion stops — after the merge, in production.
• No live state. The repo tells you what should be deployed. It says nothing about what’s actually running on each host right now, or whether the last push even landed.
• No drift detection. If someone hand-edits a host, the repo doesn’t know. Your source of truth and your reality have silently diverged, and nothing tells you.

Git gives you the desired state. Observability GitOps also needs the applied state, and a way to compare them — which is exactly what a control plane adds on top.

A git-backed control plane: edits become commits

The model that closes those gaps: keep git as the source of truth, but put a control plane in front of it that both authors and applies config.

Every edit you make in the UI becomes a commit — the config is stored in git, versioned and reviewable, before it’s rendered, validated, and published to the fleet. You get the whole code workflow — propose a change, review it, publish it, and roll back to any previous commit if it misbehaves — without anyone hand-writing YAML per host.

The validation step is the piece plain GitOps misses: because the control plane renders the actual collector config, it can reject an invalid pipeline before it ships, instead of discovering the typo when a collector won’t start. (If you’re new to how config reaches collectors, OpAMP Explained covers the delivery mechanics.)

Who changed what, when

Once configuration is git-backed, the audit trail comes for free — and it should be a first-class feature, not a raw commit log you have to git blame through. Every change is attributable: who made it, when, and exactly what changed, with a readable diff.

That matters most in the two moments you least want to be guessing: during an incident (“what changed right before this broke, and can we revert it?”) and during a review (“who touched the redaction rules last quarter?”). A version history that answers those in one click turns config changes from a source of anxiety into a routine, reversible operation.

Drift detection: desired vs applied

This is the capability that separates real observability GitOps from a repo of YAML. Because each collector reports its effective configuration back to the control plane, the platform can continuously compare what git says should be running against what each host is actually running — and flag the difference.

So when someone hand-edits a host at 2 a.m., or a push half-lands, you don’t find out weeks later when data goes missing. The fleet view shows the host as drifted, you see the diff, and re-applying the committed config brings it back in line. Desired state and reality stay converged — which was the entire point of GitOps in the first place.

You can treat drift like any other alert: surface it, decide whether the hand-edit was a legitimate fix that should be committed back or an accident to revert, and reconcile in one action. Either way, the git history stays the single record of truth for what the fleet is supposed to run — no more archaeology to reconstruct what changed.

Observability as code, for real

Treating collector config as code isn’t just tidiness — it’s how you make changes to a fleet safely and reversibly. The full version needs four things a plain repo can’t give you alone: validation before publish, live state from every host, an attributable history, and drift detection that compares desired to applied.

That’s the model LinkMesh is built on: git-backed configuration, a commit-and-publish workflow, one-click version history, and drift detection across the fleet. Compose a pipeline, commit it, publish it, and know — provably — what every collector is running. Stand one up at linkmesh.io/install.