Skip to content

Add a collector extension

An OpenTelemetry Collector’s extensions: block holds everything that isn’t a receiver, processor or exporter: authenticators, disk storage for queues, health checks, forwarders, observers. LinkMesh generates the collector’s whole config, so you don’t paste an extensions: block anywhere. You declare each extension instead:

  • You give a label (your short handle for it), the extension’s type (oauth2client, file_storage, health_check…) and its settings as YAML. You can declare any type your collector contains.
  • LinkMesh gives it a component name, adds it under extensions:, and lists it in service.extensions.
  • Your components reference it as ${ext:<label>} in a custom body, or a destination picks it in its Authentication extension field. LinkMesh fills in the name it chose.

The reason is that the collector loads its config as one document. An extension with a name nothing points to, a reference to a name that doesn’t exist, or an extension that is defined but not listed in service.extensions doesn’t fail on its own. The collector rejects the entire config, and every source and destination on it stops. When LinkMesh does the naming and the wiring, those mistakes can’t happen.

You declare an extension labelled health of type health_check, with these settings:

endpoint: 0.0.0.0:13133
path: /health

The collector receives:

extensions:
  health_check/ext_4b1e09c2:
    endpoint: "0.0.0.0:13133"
    path: "/health"
service:
  extensions: ["health_check/ext_4b1e09c2"]

The part after ext_ comes from the extension’s ID. It stays the same for the extension’s whole life. The names LinkMesh gives its own extensions (for example the storage behind Durable Delivery) never start with ext_, so yours can’t collide with them.

  1. Open the collector (Collectors, then click it) or the collector group (Collector Groups), and go to its Extensions tab. Click Add extension.

  2. Label — lowercase letters, digits, - and _, up to 32 characters, starting with a letter or digit. It must be unique on each collector, counting the collector’s own extensions and its groups’ extensions together. The form shows the token it gives you, such as ${ext:idp}.

  3. Type — the extension type exactly as the collector names it, with no /name part. The picker offers the extension types this collector contains, and says where that list came from. See The inventory check.

  4. Settings (YAML) — what goes under the extension’s entry. Leave out the extensions: key and the type/name: line. LinkMesh adds both, and a body that includes either is refused. Some extensions, such as pprof and zpages, run on their defaults and need no settings at all.

  5. Click Add extension. LinkMesh re-applies the config of every collector the extension applies to.

An extension declared on a group applies to every current member and to every collector that joins later. On a member’s own Extensions tab it is marked from group — edit it on the group.

After you save, the settings are hidden. Reveal shows them, and Replace clears the field so you can enter new settings.

Through the API it is POST /api/v1/collectors/{collectorId}/extensions (or /api/v1/collector-groups/{groupId}/extensions) with {"label": "…", "type": "…", "body": "…"}. The response carries the componentId LinkMesh chose, and a notices list when the type couldn’t be checked. PUT and DELETE on …/extensions/{extensionId} edit and remove it.

A collector can only load an extension that is compiled into its binary. So LinkMesh checks the type against the collector’s component inventory when you save. It uses the first of these that it has:

Inventory Where it comes from Shown as
Reported The collector sent the list of its components when it connected reported by collector
Known list LinkMesh ships the component list for each otelcol-contrib version it supports, and uses it for a collector that hasn’t reported one known list for otelcol-contrib 0.153.0 (its type and version)
Unknown The collector hasn’t reported its components, and LinkMesh has no list for its exact version unknown

What happens to the save:

  • The type is in the inventory — it is saved.
  • The type is not in the inventory — the save is refused. The message names the collector, its version, which inventory was used and the missing type.
  • The inventory is unknown — it is saved, with a Not checked notice. “Not checked” means just that: nothing confirmed that the binary contains the type, and nothing ruled it out either. If the binary doesn’t have it, the collector rejects its whole config. The picker accepts any type you type in, for this reason.

On a group, the save checks every member. The picker marks a type that some members lack (Missing on: …), and the save is refused while any member lacks it. A collector that joins a group is checked against the group’s extensions too, and the join is refused if it lacks one of them or already declares an extension with the same label.

The check runs again before every config push, against the collector’s current inventory. That covers the cases a save can’t see: a collector whose binary was replaced by a build without the extension, or a group member running a different build from the others. See When a collector goes incompatible.

Before LinkMesh sends a config to a collector, it checks every declared extension in that config against the collector’s inventory, the same way the save does. If one is missing:

  • The config is not sent. A collector rejects its whole config over one component it doesn’t have, so sending it would stop every source and destination on the collector.

  • The last good config keeps running. The collector goes on with the config it last applied, and keeps delivering telemetry with it. Changes you make in the meantime don’t reach it.

    This holds as long as that last config doesn’t use the missing extension itself. If a collector’s binary is replaced while the config it runs already contains the extension, the collector can’t load its own config. LinkMesh still marks it incompatible and sends nothing new, but the collector reports an error until it runs a build with the extension again, or you remove the extension.

  • The collector is marked incompatible. Its detail page shows an Incompatible chip next to its name and a red notice on every tab, naming the missing extension: its label, its type, its component name, and whether it was declared on the collector or on its group. The Extensions tab shows the same notice above the extension list, and the Events tab records collector_incompatible.

  • Only that collector is affected. In a group, each member is checked on its own. The members that have the extension get the new config as usual.

An unknown inventory is not a reason to withhold a config: the push goes ahead, as the save does.

Right after a collector connects with a changed binary, LinkMesh waits up to a minute for it to report its new component list, and checks against that rather than the old one.

The mark clears by itself, and the new config is sent, as soon as the check passes. That happens when:

  • the collector reports an inventory that contains the extension (after you go back to a build that has it), or
  • you remove the extension, or change its type to one the collector has.

The Events tab then records collector_compatible.

In the body of a custom source or destination, write ${ext:<label>} wherever the component expects an extension name:

auth:
  authenticator: ${ext:idp}
sending_queue:
  storage: ${ext:queue}

LinkMesh replaces each token with the extension’s name when it generates the collector’s config. The rules:

  • The token is the whole value. ${ext:idp} works. prefix-${ext:idp}, or a token used as a key, is refused. In a flow list, quote it: watch_observers: ["${ext:k8s}"].
  • Extension settings take a token, never a name. auth.authenticator, any storage key and watch_observers must hold ${ext:…} tokens. Anywhere else in the body, a value shaped like an extension name (<extension type>/<name>, such as oauth2client/idp) is refused too. LinkMesh chooses the names itself, so a name you type would match nothing.
  • The label must be declared where the body runs. Attaching the source or destination is refused if the label isn’t declared on any of the collectors you attach it to. On a group where only some members have the label, the members without it don’t get that source or destination. Their config is still sent, without it, and the apply reports a warning naming the label.
  • An auth.authenticator token must name an authenticator. The extension it resolves to must be able to authenticate that component — see Which extensions can authenticate. Attaching a body whose token resolves to anything else is refused with extension_not_authenticator.
  • A referenced extension can’t be removed or relabelled. Removing or relabelling it is refused, and the message lists the sources and destinations that still reference it — by token, or through their Authentication extension field. Remove the reference first. Changing the type of an extension used as an authenticator to a type that can’t authenticate that component is refused the same way, with extension_not_authenticator.

The Raw YAML editor has a Reference an extension picker that inserts the token at the cursor. Before you save, the editor flags a label that isn’t declared anywhere, a token inside a longer string, and a malformed label.

An extension’s own settings can’t reference another declared extension. Tokens work in custom source and destination bodies. A destination built from the form (not a custom one) takes an authenticator through its Authentication extension field instead — see Authenticate a destination through an extension.

Authenticate a destination through an extension

Section titled “Authenticate a destination through an extension”

A destination you create from the form — OTLP gRPC, OTLP HTTP, Loki, Grafana Cloud, Prometheus RW, Datadog, Splunk HEC or Elasticsearch — has an Authentication extension field. Pick a declared label there, and the destination’s exporter authenticates through that extension:

exporters:
  otlphttp/output_dst_1c9e0d4a:
    endpoint: "https://otlp.example.com"
    auth:
      authenticator: "oauth2client/ext_4b1e09c2"

The rules:

  • One authenticator per destination. A destination with basic_auth credentials (or, for Grafana Cloud, an instance ID and API token) can’t also have an authentication extension: the exporter would use one and silently ignore the other. The field is unavailable while those credentials are set, and a save that sets both is refused with auth_extension_credentials_conflict. Clear one of them.
  • The label must be declared where the destination runs — the same rule as for a token. Attaching the destination is refused with extension_reference_unresolved if no collector you attach it to has the label. On a group, members without it don’t get the destination, and the apply reports a warning.
  • The extension must be an authenticator for this destination. The field offers only declared extensions whose type can authenticate the destination’s exporter — see Which extensions can authenticate. Attaching the destination, or pointing an attached one at such a label, is refused with extension_not_authenticator, naming the label and its type.
  • Kafka, File and Debug destinations don’t have the field. Their exporters have no authenticator setting.
  • Custom destinations don’t have it either. Their body names the authenticator itself: auth: {authenticator: ${ext:<label>}}.

Through the API, the field is auth_extension in the destination’s defaultConfig (or in an activation’s configOverrides), holding the label. GET /api/v1/extension-labels?authenticatorFor=<destination type> lists the labels the field may name for that destination type.

An exporter’s or receiver’s auth.authenticator accepts only an extension of a type that can authenticate that component. Anything else — a health_check, a file_storage, an http_forwarder — passes the collector’s config check and then stops the whole collector when it starts, so LinkMesh refuses it. There is no override: a type not listed here is refused, even if your collector build contains it. Use a custom destination without an authenticator, and set the credentials in its body, for anything else.

Where Permitted extension types
OTLP gRPC destination (and custom otlp, otelarrow, stef, loadbalancing, coralogix bodies) asapclient, basicauth, bearertokenauth, googleclientauth, headers_setter, oauth2client
Every other destination with an authenticator (OTLP HTTP, Loki, Grafana Cloud, Prometheus RW, Datadog, Splunk HEC, Elasticsearch, and custom bodies of other exporters) the six above, plus azure_auth, sigv4auth, sumologic
A custom source body (a receiver’s auth.authenticator) azure_auth, basicauth, bearertokenauth, oidc

azure_auth, sigv4auth and sumologic authenticate HTTP requests only, so a gRPC exporter can’t use them. oidc validates incoming requests, so it works only on a receiver. For basicauth on a destination, set client_auth in its settings; on a source, htpasswd.

A destination saved with an authenticator that isn’t permitted is not sent to any collector: the collector gets the rest of its config without that destination, and the apply reports a warning naming the destination, the extension and its type. Point the destination at a permitted extension, or clear the field, and apply again.

All four run on otelcol-contrib as LinkMesh ships it. Replace the hosts, ports and paths with your own.

OAuth2 client credentials on an OTLP/HTTP destination

Section titled “OAuth2 client credentials on an OTLP/HTTP destination”

Declare the authenticator on the collector:

  • Label idp, Type oauth2client

  • Settings:

    client_id: linkmesh
    client_secret: ${env:IDP_CLIENT_SECRET}
    token_url: https://idp.example.com/oauth2/token
    scopes: ["api.write"]

Then create the destination. From the form:

  1. Destinations, then New Destination. Keep Guided mode and pick Exporter Type OTLP HTTP.
  2. Set Endpoint to https://otlp.example.com. Leave basic_auth empty.
  3. In Authentication Extension, pick idp, and create the destination.

Or, as a custom destination, with exporter type otlphttp and this body:

endpoint: https://otlp.example.com
auth:
  authenticator: ${ext:idp}

Either way, the exporter fetches a token from token_url and sends it as a bearer token. A wrong secret fails this exporter only. The collector’s other destinations keep delivering, and the collector’s log shows the authentication error against this one exporter. See Keep secrets out of LinkMesh for setting IDP_CLIENT_SECRET on the host.

A custom destination with a disk-backed queue

Section titled “A custom destination with a disk-backed queue”

Declare the storage:

  • Label queue, Type file_storage

  • Settings:

    directory: /var/lib/otelcol/queue
    create_directory: true

and point the exporter’s queue at it:

endpoint: https://otlp.example.com
sending_queue:
  storage: ${ext:queue}

Anything waiting in the queue is kept on disk and delivered after the collector restarts. The directory must be writable by the user the collector runs as.

The simpler way to get the same result is the Durable Delivery setting on the destination itself, with no extension to declare. See Resume after restart. Use one or the other on a destination, not both.

An extension doesn’t have to be referenced by anything. http_forwarder accepts HTTP on one address and forwards it to another:

  • Label forward, Type http_forwarder

  • Settings:

    ingress:
      endpoint: 0.0.0.0:6060
    egress:
      endpoint: http://upstream.example.com:4318

LinkMesh doesn’t check the listen port against the ports your sources use. Pick one that nothing else on the collector binds. If the port is taken, the collector fails to start.

  • Label health, Type health_check

  • Settings:

    endpoint: 0.0.0.0:13133
    path: /health

GET http://<collector-host>:13133/health then returns 200 while the collector is running. Point a load balancer or a Kubernetes probe at it.

Runtime Declared extensions
otelcol-contrib (OpAMP) Any extension type in the collector’s inventory, as described above
Grafana Alloy None. Every type is refused

Grafana Alloy has no generic extension block. Its authenticators and storage are components with their own settings, written in River rather than YAML, and an OpenTelemetry extension body doesn’t translate into River. So a declaration on an Alloy collector is refused, not converted or dropped. The message ends:

declared extensions render on the otelcol runtime only — Grafana Alloy has
no generic extension block, and an OpenTelemetry extension body does not
translate into River. Declare it on an otelcol collector instead

On an Alloy collector the Extensions tab says so and offers no Add extension button. A group with any Alloy member can’t take a declared extension, and an Alloy collector can’t join a group that declares one. Likewise, a destination with an Authentication extension can’t be attached to an Alloy collector, and an Alloy collector can’t join a group that has such a destination.

Write credentials as ${env:NAME}. The collector reads the variable from its own environment when it starts, so the value never leaves the host. LinkMesh stores and shows ${env:NAME} exactly as you wrote it.

A credential written in clear is not stored in clear. When you save, every setting whose key looks like a credential is moved into LinkMesh’s secrets vault, and the stored settings hold a ${secret:…} reference instead. Keys containing secret, password or token count, among others. An endpoint address such as token_url is not a credential and is stored as you wrote it, unless the URL itself carries one (a user and password, or a query string); then it is moved into the vault too. If the vault is disabled, the save is refused rather than storing the value in clear.

Where the collector’s environment is set

Section titled “Where the collector’s environment is set”

The supervisor starts the collector, and the collector inherits the supervisor’s environment. So set the variable on the supervisor.

Linux (install script or manual install). The supervisor runs as the systemd unit opamp-supervisor. Add a drop-in rather than editing the unit file, because re-running the installer rewrites the unit:

sudo systemctl edit opamp-supervisor
[Service]
Environment="IDP_CLIENT_SECRET=…"

Or, to keep the value out of the unit, use EnvironmentFile=/etc/otelcol-supervisor/secrets.env and make that file readable by root only. Then restart:

sudo systemctl restart opamp-supervisor

Kubernetes (DaemonSet). Add the variable to the supervisor container’s env:, taken from a Kubernetes Secret:

env:
  - name: IDP_CLIENT_SECRET
    valueFrom:
      secretKeyRef: { name: idp-credentials, key: client-secret }

The pods pick it up when they restart. See Install on Kubernetes for the full manifest.

A variable that isn’t set resolves to an empty string. For an authenticator that usually means the exporter fails to authenticate, not that the collector fails to start. Set it before you reference it.