Secrets in Collector Config

Every useful OpenTelemetry Collector config ends up needing a secret. The moment you add an exporter, you need a credential to authenticate to the destination: a Datadog API key, a Splunk HEC token, a Grafana Cloud username and access policy token, a client certificate for mTLS to your own gateway. The Collector config is just YAML, and the obvious place to put that key is right there next to the exporter that uses it — which is exactly the mistake.

The problem isn’t that the Collector can’t hold a secret. It’s that Collector config belongs in version control, and version control is the one place a plaintext credential must never land. This post is about resolving that tension: how to keep secrets out of the YAML you commit while still handing the running Collector the credentials it needs, how to rotate them without downtime, and where the honest limits of the Collector’s built-in handling are.

Where secrets show up in a Collector config

Secrets cluster in a few predictable places, almost all of them in exporters and receivers that talk to something authenticated:

• Backend API keys and tokens — Datadog api::key, Splunk HEC token, an otlphttp exporter with a bearer token in headers, Grafana Cloud basic-auth.
• TLS material — client certificates and private keys for mTLS, CA bundles, the tls::key_file and tls::cert_file paths on both exporters and receivers.
• Upstream credentials — a Kafka receiver or exporter with SASL username and password, a Prometheus remote-write endpoint behind auth.

Every one of these is a value that grants access to a system. The config that references them is something you want in git — reviewed, versioned, diffable — for all the reasons any infrastructure-as-code artifact belongs there. That is the whole GitOps tension in one sentence: the config must be committed; the secrets must not be.

The rule: never hardcode, never commit

Hardcoding a secret into committed YAML is worse than it looks, because git does not forget. Once a key lands in a commit it is in the history on every clone, every fork, every CI cache, and every developer laptop — and rotating it out of the current file does nothing about the twenty places the old value still lives. A leaked credential in git history is a rotate-everything incident, not a git revert.

So the discipline is absolute: the committed config never contains a plaintext secret. What it contains instead is a reference — a placeholder that gets resolved to the real value at the moment the Collector starts, from a source that is not your git repo.

Environment-variable substitution: the built-in primitive

The Collector’s native mechanism for this is substitution. Anywhere in the config you can write ${env:VAR_NAME} and the Collector expands it from the process environment at load time — and the same machinery resolves ${file:/path/to/secret} from a mounted file, which pairs naturally with Kubernetes Secrets or a Vault CSI volume. Your committed YAML holds only the placeholder:

exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
  otlphttp/vendor:
    endpoint: https://otlp.vendor.example
    headers:
      authorization: ${env:VENDOR_BEARER_TOKEN}

The real DD_API_KEY never appears in the file. It has to be present in the Collector’s environment when the process starts, and that is where the rest of the platform does the work. This is deliberately basic — it is string substitution, not a secrets system. The Collector trusts that whatever set the environment variable did so securely, and offers nothing itself for storage, rotation, or access control. That honesty matters: you compose the Collector with a real secret manager; you do not rely on the Collector to be one.

Kubernetes Secrets as environment variables

On Kubernetes, the common pattern is to store the credential in a Secret and project it into the Collector container as an environment variable, which the ${env:...} syntax then picks up:

env:
  - name: DD_API_KEY
    valueFrom:
      secretKeyRef:
        name: otel-backend-credentials
        key: dd-api-key

Now the config in git says ${env:DD_API_KEY}, the Deployment references a Secret by name, and the actual value lives only in the Kubernetes Secret — managed with its own RBAC, ideally backed by encryption at rest. Be clear-eyed about the limits: a stock Kubernetes Secret is base64-encoded, not encrypted, and anyone who can read Secrets in the namespace can read it. Enable encryption at rest, lock down RBAC, and for stronger guarantees reach for an external manager.

External secret managers and injection

For real rotation, auditing, and least-privilege access, the credential lives in a dedicated secret manager — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault — and gets injected into the Collector’s environment at deploy time. Two common shapes:

• Sync into a Kubernetes Secret. Tools like the External Secrets Operator pull from Vault or a cloud manager and materialize a Kubernetes Secret, which you then mount as env vars exactly as above. The source of truth is Vault; the Secret is a short-lived projection.
• Sidecar/agent injection. A Vault Agent (or CSI driver) fetches the secret and writes it to a mounted file (which the config can read directly with ${file:...}) or the process environment, so it never touches etcd at all.

Either way the Collector still just reads ${env:DD_API_KEY} — the sophistication is entirely in how that variable got populated. The Collector doesn’t know or care whether the value came from a static Secret or a freshly-minted Vault lease, which is precisely why this composition is clean.

Rotating secrets without downtime

Rotation is where hardcoded secrets hurt most and referenced secrets shine. Because the Collector reads its credential from the environment at startup, rotation is: update the value in the secret manager, then restart the Collector so it picks up the new one. A few practical notes:

• Roll, don’t stop. In Kubernetes, a rolling restart of the Collector Deployment or DaemonSet replaces pods one at a time, so telemetry keeps flowing while each pod picks up the rotated value. Nothing goes dark.
• Overlap the old and new. If the backend supports two active keys, add the new key before revoking the old one. Roll the fleet onto the new value, confirm exports succeed, then retire the old key. This avoids the window where a half-rolled fleet is authenticating with a key you already killed.
• The config never changes. The committed YAML still says ${env:DD_API_KEY} through the whole rotation. That is the point — rotation touches the secret store, not the versioned config, so it leaves no diff and needs no config review.

The Collector does not hot-reload environment variables mid-process, so plan on a restart being part of any rotation. That is a mild constraint, not a real one, given rolling restarts are cheap.

Where a control plane fits

Managing this across a fleet by hand is doable but fiddly, and it’s easy for one node’s config to drift into holding a literal key “just to test something.” This is where a control plane earns its place: with LinkMesh you enter a backend credential once, centrally, on the destination that uses it — nobody hand-pastes a key into the YAML on an individual node. Rotation stops being a fleet-wide scavenger hunt: update the credential on the destination and every collector that ships to that backend picks up the new value automatically, no per-node edit.

The part that matters for the tension this post is about: LinkMesh does the same reference-not-value trick as ${env:...}, but end to end and server-side. A credential lives as a named secret in an encrypted store (AES-256-GCM, the key is yours and never leaves your database boundary), and the destination config references it by name — ${secret:grafana-cloud-token}. That reference is what gets committed to git, returned by the API, and shown in a diff; the plaintext value is never any of those things. The real value is substituted only at delivery — the moment the config is pushed to a collector over the authenticated channel — so the running collector gets exactly the bytes it needs while your GitOps repo, your config history, and a PVC backup all hold nothing but the reference. Rotate the secret once and LinkMesh re-renders and re-delivers to every collector that references it.

That keeps the properties you actually want: every config change is versioned and auditable (the model behind GitOps for collector config), access to the managed config is permission-controlled, credentials are redacted on every read path, and — because LinkMesh is self-hosted and pipeline telemetry never flows through it — your data and your credentials stay on your own infrastructure. It composes with the same enforcement model that keeps every other policy consistent across the fleet.

Where to start

You don’t need Vault on day one. The progression that works:

1. Stop committing secrets now. Replace every literal credential in your config with ${env:VAR} and move the value into the environment. Then scrub git history if anything already leaked.
2. Back it with Kubernetes Secrets (encryption at rest on, RBAC tight) so the value has a managed home, not a shell export.
3. Graduate to an external manager — Vault or a cloud secret manager with injection — when you need rotation, auditing, and least-privilege access.
4. Reference by name centrally so no node’s committed config ever holds plaintext, and rotation touches the secret store rather than the pipeline.

The Collector’s own secret handling is deliberately thin: ${env:...} and ${file:...} substitution and nothing more. That is fine, as long as you remember it’s a reference mechanism and put a real secret manager behind it. Keep the plaintext out of git, resolve it at startup, rotate it in the store — and your most sensitive credentials never ride along in the config that describes your pipeline.