LinkMesh Server 1.342.8

Breaking changesReleased 31 August 2026

Availability

This release is no longer available to download.

The files for this release are gone and cannot be rebuilt, so we can no longer honestly offer it.

Upgrade to 1.350.4 or a later release; those remain available.

Withdrawn on 21 September 2026. What it changed is kept below, so you can still read your way forward from a version you are running.

Breaking changes

This release contains breaking changes. Read the upgrade notes below before upgrading.

Read the marker, not the version number. Version numbers do not carry a semver major warning: a breaking change can ship in a minor release. Rely on the breaking-change marker, not on the version number.

The internal certificate authority and every client-certificate endpoint, config key and permission is gone. Collectors authenticate with a Bearer token over TLS, and nothing else.

Written by us

Before you upgrade

WHO IS AFFECTED. You are affected if any of these is true: you set a ca: block or the CA_SERVER_SECRET environment variable in your server configuration; you have scripts or monitoring that call /api/v1/certificates/... or /api/v1/ca/...; or you built a custom role that grants any certificates:* permission. If none of those applies, this changes nothing for you and your collectors keep connecting exactly as before.

WHAT CHANGED, precisely.

• The whole ca: configuration block is removed: ca.serverSecret (and its CA_SERVER_SECRET environment binding), ca.keySize, ca.validityYears, and ca.subject.CN / .O / .OU. They are no longer read. Leaving them in your config file is harmless but does nothing.
• These HTTP endpoints no longer exist and now answer 404: everything under /api/v1/ca/, everything under /api/v1/certificates/ including GET /api/v1/certificates/chain.pem, the certificate listing, and POST /api/v1/certificates/renew.
• The Certificates section is gone from the web interface.
• The certificates:read, certificates:update and related permissions are removed from the built-in roles, and a one-shot migration at first boot strips them from any custom role that still carries them.

UPGRADE STEPS.

1. Before upgrading, search your server configuration for a ca: block and for CA_SERVER_SECRET in your environment, systemd unit or Kubernetes manifest. Delete them. This is tidy-up, not a prerequisite — an unread key breaks nothing — but leaving them in place will mislead whoever reads that file next.
2. Search your own tooling for the string /certificates and /ca/. Anything calling those paths must be removed; there is no replacement endpoint, because there is no longer anything to issue, list or renew.
3. Upgrade and let the server start once. The role migration runs at boot and needs no input.
4. After the first boot, open Access Control and check any custom roles you maintain. A role whose only purpose was certificate administration will now grant nothing; delete it or repurpose it deliberately rather than leaving an empty role in the list.

WHAT HAPPENS IF YOU SKIP THIS. Nothing breaks at upgrade time, and no collector drops off — this is the important part, because the change sounds far more alarming than it is. Your collectors already authenticate with the durable Bearer token and continue to do so untouched. What fails is tooling: a monitoring check or script that polls a certificate endpoint starts returning 404, and depending on how it is written it will either alert or fail silently. That is the failure to look for, and it will not announce itself as being related to this upgrade.

WHY THIS CHANGED, because the removal of a security mechanism deserves an explanation. Server-pushed client certificates were built to the OpAMP specification's model — bootstrap with a token, receive a certificate over the live session, reconnect with mTLS. Lab testing established that the canonical opampsupervisor (v0.23.0) does not advertise support for receiving connection settings, silently ignores a certificate offer, and worse, stalls its own status and heartbeat loop within about six seconds of receiving one: the collector drops offline while systemd still reports it connected. This was reproduced four times, across first enrolment, re-enrolment and rotation. So the machinery was not a working security feature being withdrawn; it was a feature that could take a healthy collector offline, and it is removed for that reason. The durable Bearer token over TLS terminated at the ingress is the steady-state credential for both Alloy and OpAMP collectors, which is how OpAMP fleets are run in practice.

The trade-off, stated plainly: there is no per-collector, individually revocable transport credential. Revocation is by enrollment token or by deregistering the collector, not by certificate serial. If you need genuine mTLS, it must be provisioned out of band at install time (cert_file / key_file in supervisor.yaml); the server-side reader for that path is kept and inert, so a collector that presents a client certificate at connect time still authenticates end to end. That path is not built and not supported.

Shipped in v1.267.1, 2026-07-10.

Enrollment tokens are shown once, at creation. The token list no longer returns the token itself, and the plaintext of every existing token is erased from the database at first boot.

Written by us

Before you upgrade

WHO IS AFFECTED. Anyone who reads an enrollment token back out of the interface or the API when enrolling a new collector — rather than keeping it from the moment it was created. If your enrolment runbook says "go to Settings, copy the token, paste it into the install command", that runbook stops working. Your already-enrolled collectors are not affected and do not need to be touched.

WHAT CHANGED. Enrollment tokens were stored in plaintext and returned in full on every list call. They are now stored as a SHA-256 hash, exactly as API tokens and OTLP tokens already were. The raw value is returned once, in the response to the call that creates it, and never again. The list shows only the first eight characters as a non-secret handle, plus the metadata.

A one-shot migration runs at first boot: it computes the hash for every existing token and erases the stored plaintext.

UPGRADE STEPS.

1. Before upgrading, decide whether you need any of your current enrollment tokens in readable form. If you do — for example it is pasted into a configuration-management template, a runbook, or a password manager entry you have not filled in yet — copy it now. After the upgrade it cannot be recovered from the server by any means, including from the database directly.
2. Upgrade and let the server start once. The migration is automatic, idempotent, and needs no input.
3. Verify the fleet is still connected. Every collector holding a raw token continues to authenticate, because the token it presents hashes to the value the migration backfilled. This was the specific risk in this change and it is explicitly covered by tests, but it is worth confirming in your own environment: the collector list should show the same number of connected collectors as before the upgrade.
4. Update any runbook or template that says to copy a token out of the token list. The new shape is: mint a token, copy it immediately from the creation response, store it wherever you keep secrets.

WHAT HAPPENS IF YOU SKIP THIS. Nothing fails during the upgrade, and no collector is disconnected. The failure arrives later and looks like a documentation problem rather than an upgrade consequence: somebody follows the enrolment runbook, opens the token list to copy the token, and finds only a short prefix. The fix at that point is simply to mint a new token — no existing collector is harmed by doing so — but the person hitting it will not know that, and will be doing it in the middle of a task that has just stopped working.

WHY THIS CHANGED. The enrollment token is the only credential a collector presents on every reconnect. Storing it in plaintext and echoing it on every list meant that read access to the token list, or to a database backup, was equivalent to the ability to enrol a collector into your fleet.

Shipped in v1.305.0, 2026-07-21.

Plaintext credentials sitting in your committed GitOps configuration are rewritten to placeholders at first boot. Anything the server cannot recover from the live entity is skipped and needs your attention, and the old values remain in git history regardless.

Written by us

Before you upgrade

WHO IS AFFECTED. Anyone whose destination or source configuration contains an inline credential — a password, an API key, a bearer token typed directly into a field rather than entered as a ${secret:name} reference. If every credential you use is already a secret reference, this changes nothing.

WHAT CHANGED. The GitOps working clone used to hold live credentials in plaintext at its tip, because a destination's activation snapshot was committed with whatever value was in the field. A one-shot migration at first boot rewrites those committed snapshots so the credential reads as a masked placeholder instead.

Delivery is safe by construction: the value that is actually sent to a collector is sourced from the live entity in the database, not from the committed snapshot, and a masked snapshot value is treated as "unchanged" during the merge. So masking the snapshot does not change what your pipelines deliver.

There is one case the migration deliberately refuses to touch, and it is the one that needs you: an activation whose credential exists ONLY in the snapshot, with no live entity behind it to recover the value from. Masking that would silently break delivery, so it is skipped and a warning is logged instead.

UPGRADE STEPS.

1. Upgrade and let the server start once. The migration is idempotent and best-effort; a failure on one activation is logged and never stops the boot.
2. Read the server log from that first boot and search for the migration's warnings. Each one names an activation it skipped because the credential was not recoverable from a live entity.
3. For every skipped activation, do one of two things: convert the credential to a ${secret:name} reference using the Secrets section, or rotate the credential at the destination and enter the new value. Either resolves it. Leaving it is also a decision — the plaintext simply stays in the working clone, which is where it was before this release.
4. Now consider rotation for everything else. This is the step people skip and it is the one that matters. The migration masks the TIP of the git history. It does not and cannot rewrite history: every credential that was ever committed is still readable in an earlier commit of that repository. If your GitOps repository is remote, shared, or backed up, treat those credentials as exposed and rotate them. If you skip this, the honest description of your state after the upgrade is "the tip is clean and the history is not".

WHAT HAPPENS IF YOU SKIP THIS. The upgrade completes and your pipelines keep delivering. What you miss is the two things the migration cannot do for you: the skipped activations sit in the log unaddressed, and the credentials in git history stay readable. Neither produces an error, an alert or a visible symptom — which is precisely why this note exists.

Shipped in v1.309.0, 2026-07-22, as part of the secrets work that also introduced the Secrets section, secret references and rotation.

A collector group now holds one kind of collector. Adding a VM collector to a Kubernetes group, or the reverse, is refused with 422.

Written by us

Before you upgrade

WHO IS AFFECTED. Anyone whose collector groups mix VM or host collectors with Kubernetes collectors, and anyone whose automation adds collectors to groups. If you run only VM collectors, or only Kubernetes collectors, this does not affect you.

WHAT CHANGED. A collector's kind is derived from whether it enrolled through a reusable fleet token: a collector that carries a fleet id is kubernetes, everything else is vm. Both membership-mutation paths now check that kind against the group's and refuse a mismatch:

• POST /api/v1/collector-groups/{id}/collectors returns 422 with a message naming both kinds.
• A PATCH that sets collectorIds is checked the same way.

Fleet-token auto-join is unaffected, because a fleet collector joins its own fleet's Kubernetes group and is consistent by construction. Adoption and discovery do not attach collectors to groups, so they are unaffected too.

This guards the mutation paths only. An existing mixed group is not torn apart by the upgrade and keeps working.

UPGRADE STEPS.

1. Before upgrading, list your collector groups and check whether any holds both kinds. A group with a mix will keep functioning, but you will not be able to change its membership afterwards without splitting it.
2. Split any mixed group into a VM group and a Kubernetes group, and attach the routes and configuration each kind needs. Do this before the upgrade if you can, because afterwards the only way to move a collector out of a mixed group is to remove it and add it to a new one — which the guard permits, but which takes more steps than planning the split.
3. Check any automation that adds collectors to groups and make sure it handles a 422. A script that assumes a 2xx will now fail, and the message tells you exactly which kind it tried to add to which.

WHAT HAPPENS IF YOU SKIP THIS. Nothing at upgrade time. Your mixed group keeps running with the configuration it already has. The failure arrives the next time somebody tries to add a collector to it — probably during an unrelated task, probably from a script, and the 422 will read as a permissions or validation problem rather than as a rule that was introduced here.

WHY THIS CHANGED. A group's whole purpose is that its members share one configuration. A VM collector and a Kubernetes DaemonSet collector have genuinely different receiver topologies — different sources, different discovery, different semantics — so a config that suits one cannot suit the other. A mixed group therefore could not produce a correct configuration for all its members; the mismatch simply surfaced later, as a failed apply at the edge rather than as a refusal at the point where somebody made the mistake.

Shipped in v1.324.0, 2026-07-23.

The PII and credit-card masking templates changed their underlying processor type, and any blocked_values or allow_all_keys settings you had overridden on a masking processor are dropped rather than translated.

Written by us

Before you upgrade

WHO IS AFFECTED. Anyone using a processor created from the pii-redaction or cc-masking template, and in particular anyone who edited such a processor's configuration to add or change blocked_values or allow_all_keys. If you use those templates with their default configuration, the migration handles it and you have nothing to do.

WHAT CHANGED. Both templates were built on the redaction processor type, configured with allow_all_keys and blocked_values. They are now expressed as a transform processor with OTTL statements. The template definitions re-seed themselves at boot, so the templates migrate without help.

What does not migrate itself is your own stored per-processor override. Configuration generation merges your override map on top of the template's default, so a leftover blocked_values / allow_all_keys pair would be rendered inside a transform block, where those keys mean nothing. Those stale keys are therefore dropped, not translated.

The decision not to translate them was deliberate: the two shapes are not equivalent. The redaction processor took a list of value patterns; transform takes OTTL statements. A mechanical translation would have produced something that looked like your configuration and masked different things — which is worse than dropping it, because you would not know to check.

UPGRADE STEPS.

1. Before upgrading, list the processors in your pipelines that were created from pii-redaction or cc-masking, and for each one record whether you customised its configuration. The values you need to note are blocked_values and allow_all_keys.
2. Upgrade and let the server start once.
3. For each processor you noted in step 1, open it and re-express your customisation as OTTL statements in the transform configuration. The template's own statements show the form to follow.
4. Use processor preview on a representative record to confirm the values you intended to mask are still masked. This is the only check that actually tells you whether step 3 worked.

WHAT HAPPENS IF YOU SKIP THIS. Your pipeline deploys, the collector reports healthy, and the customisation you had added is simply not applied. Values you believed were being masked pass through unmasked. There is no error and no warning at any point, which is why step 4 is worth the few minutes.

WHY THIS CHANGED, and why it was urgent rather than cosmetic. The redaction processor type compiles to otelcol.processor.redaction, a component that no released version of Grafana Alloy contains — it is absent at 1.0.0, 1.6.0 and 1.18.0 alike, while every other processor in the catalogue is present. Alloy parses a served remote configuration as a whole and rejects the entire thing if it names a component the build does not have. So a single unknown component cost the collector every source, every processor and every destination it had, while it continued to report itself healthy. Re-expressing the templates on transform removed a component that could silently empty a whole pipeline.

Shipped in v1.328.1, 2026-08-02.

Creating or updating a destination with configuration fields at the top level of the request is now rejected with 400. It used to return 201 and silently discard them.

Written by us

Before you upgrade

WHO IS AFFECTED. Anyone who creates or updates destinations through the API with their own client or script. The web interface has always built the request correctly and is unaffected.

WHAT CHANGED. POST /api/v1/destinations and PATCH /api/v1/destinations/{id} used to ignore any field they did not recognise. A request that put endpoint, compression, tls and so on at the top level — instead of inside defaultConfig — returned 201 Created with every one of those fields thrown away. The destination was created with no configuration at all.

Now:

• Unknown top-level fields are rejected with 400, and the message names each offending field individually. Where the field is recognisably a configuration field, the message also tells you that defaultConfig is where it belongs.
• The set of known fields is derived by reflection over the model's JSON tags, so a field added later does not get rejected because somebody forgot to update a hand-written list.
• templateId is now genuinely applied rather than ignored. An explicit value beats the template's value. An unknown templateId is a 400, not a silent no-op.

UPGRADE STEPS.

1. Before upgrading, find every place your tooling creates or updates a destination and check the request shape. Configuration fields belong inside defaultConfig; anything else at the top level will now be refused.
2. While you are there, read back the destinations those scripts created. This is the important step. A destination created through the broken path has an empty configuration, which means it has been running on the schema defaults — for OTLP that is localhost:4317, and for OTLP/HTTP http://localhost:4318. In other words it has been shipping telemetry to the collector's own host rather than to the endpoint you specified. Check the defaultConfig of each one and repair anything that is empty or wrong.
3. Upgrade, then re-run your tooling and confirm it succeeds. A 400 at this point is telling you exactly which field is in the wrong place.

WHAT HAPPENS IF YOU SKIP THIS. Your scripts start failing with 400 at the next run. That is the intended outcome and it is the smaller half of the problem. The larger half is step 2: any destination already created through the old path is misconfigured right now, silently, and upgrading does not repair it. Nothing in the interface or the API flags it — a GET on such a destination simply reads as "not configured yet", and the collector reports healthy while delivering nowhere.

WHY THIS CHANGED. Strict decoding was chosen deliberately over the gentler alternatives — echoing the ignored fields back, or accepting them quietly — because the previous behaviour was an operation that reported success for work it had not done. Breaking a client that is already failing silently, loudly and at the point of the mistake, is the lesser harm.

Shipped in v1.339.3, 2026-08-19.

Grafana Alloy below 1.13.0 is no longer a supported collector version. Every Alloy collector under it is flagged Unvalidated, and the recommended version moves from 1.6.0 to 1.13.2.

Written by us

Before you upgrade

WHO IS AFFECTED. Anyone running Grafana Alloy collectors below 1.13.0. Check before upgrading: the collector list shows each Alloy collector's version. OpenTelemetry Collector (otelcol) collectors are unaffected — this is an Alloy floor only.

WHAT CHANGED. Three things, and they arrive together:

• The supported minimum for Alloy moves from 1.0 to 1.13.0. Any Alloy collector below it now carries the red Unvalidated badge where it showed nothing before. This is a fleet-wide visible change on the day you upgrade.
• The recommended version moves from 1.6.0 to 1.13.2, so every Alloy collector below 1.13.2 now shows the amber Update badge.
• POST /api/v1/collectors/{id}/upgrade defaults its target to the recommended version. A collector sitting on 1.6.0 used to get "already at recommended" (409) and now gets a real upgrade dispatched to 1.13.2. If you call that endpoint from a script, its behaviour has changed under you.

The floor constrains what the server may emit into a collector's configuration. Nothing at runtime refuses to serve a configuration to a below-floor collector, so this upgrade does not cut anyone off.

UPGRADE STEPS.

1. Before upgrading, list your Alloy collectors and their versions. Anything below 1.13.0 is what this note is about.
2. Upgrade those collectors to 1.13.2 — the newest patch of the floor's minor — before or soon after upgrading the server. Do them in whatever order suits you; there is no ordering constraint between server and collector here.
3. Upgrade the server.
4. Check the collector list. Any remaining red Unvalidated badge is an Alloy collector still below 1.13.0 and is your remaining work. An amber Update badge means merely behind the recommended version, which is a softer signal and not urgent.
5. If you script against the upgrade endpoint, re-read what it now does for a collector on an old version: it dispatches rather than refusing.

WHAT HAPPENS IF YOU SKIP THIS. Nothing breaks on the day. Your below-floor collectors keep receiving configurations and keep shipping telemetry, and the only immediate change is the badge.

The risk is later, and it is severe enough to be worth stating precisely. Alloy parses a served remote configuration as a whole. If it receives a configuration naming a component its build does not contain, it does not skip that component — it rejects the entire configuration and runs nothing: no source, no processor, no destination. On a first enrolment there is no cached configuration to fall back on. And the collector continues to report itself healthy to the server throughout.

So the day the server emits any component newer than your collector's build, that collector silently stops shipping telemetry while looking fine. That is the failure the floor exists to prevent, and being below the floor is what makes you eligible for it. If an Alloy collector stops shipping after a configuration push, check its version before anything else.

WHY 1.13.0. It is the release that added otelcol.connector.count, which per-rule masking metrics need. Established two independent ways: the upstream changelog entry under the 1.13.0 heading (released 2026-02-05), and the component reference returning 200 at tag v1.13.0 and 404 at v1.12.2, with a control component that returns 200 at both to prove the 404 was a genuine absence rather than a documentation reshuffle.

Shipped across v1.339.x–v1.340.1, 2026-08-21.

ONE KNOWN GAP, recorded rather than smoothed over: the Unvalidated badge's tooltip does not name the floor version — it points at the recommended version instead, because that is the number the API sends. So the badge tells you that you are below the floor without telling you what the floor is. It is 1.13.0, and that is why it is written here.

The cc-masking and pii-redaction processor templates have been removed. A migration re-anchors every affected processor at boot.

Written by us

Before you upgrade

WHO IS AFFECTED. Anyone whose pipelines contain a processor created from cc-masking or pii-redaction, including disabled ones. If none of your pipelines uses either, this does not affect you.

WHAT CHANGED. Both templates are gone from the processor catalogue, which drops from 29 to 27 entries. Masking is now built from the rule-based PII masking processor, which references maintained detector versions instead of storing a copy of a pattern.

UPGRADE STEPS.

1. Upgrade the server and let it start once. A migration runs at boot and re-anchors every affected processor — you do not have to edit pipelines by hand.
2. A processor that ran the template's built-in patterns becomes a PII masking processor with equivalent detector rules. It masks the same values it masked before.
3. A processor whose log_statements you had overridden becomes a Custom transform, with your statements preserved byte-for-byte. Nothing you wrote is rewritten.
4. Open Pipelines and confirm each previously-affected processor now shows as PII masking or Custom transform.
5. If one instead shows an empty Transform, it did not migrate. Do not deploy that pipeline — see below.

WHAT HAPPENS IF YOU SKIP THIS. A processor left pointing at a removed template renders as an empty transform. The configuration is valid, the pipeline deploys, the collector reports healthy — and the data passes through unmasked. This failure is silent, which is why step 4 is worth the two minutes.

Disabled processors migrate too, so re-enabling one later is safe.

Shipped in v1.341.1, 2026-08-26. Note the relationship to the earlier masking change in this same release window: that one moved the templates onto a new processor type, this one removes them altogether. If you are upgrading across both — which you are, since both are inside this release — you only need to do the work described here.

Stopped collectors now keep their licensed seat, and bringing a decommissioned collector back is refused at your limit. Your licensed collector count can go up after this upgrade.

Written by us

Before you upgrade

WHO IS AFFECTED. Anyone at or near their licensed collector limit. If your fleet has headroom, you will see a number change and nothing else.

WHAT CHANGED. Two things, and together they can make your licensed collector count go up after this upgrade:

• A collector that reports it is shutting down keeps its seat. Previously a stopping Alloy collector silently freed one, while a stopping OpAMP collector did not — the same action, two different outcomes, depending on which agent you ran.
• Bringing a decommissioned collector back into service — Reset Connection in the interface, or a re-registration — now passes the same licence check as a new enrolment. At your limit it is refused with 403 license_cap_reached.

This is a change to what counts against your licence, so read it as a licensing change rather than as a bug fix, even though it arrived in a release that carried a great many bug fixes.

UPGRADE STEPS.

1. Before upgrading, record your current count: GET /api/v1/license reports maxCollectors and currentCount.
2. Upgrade, then read the same endpoint again. A higher currentCount is expected if you run Alloy collectors that were stopped — those seats were being released silently and are now held.
3. If the new count is at or above maxCollectors, decommission what you no longer need with DELETE /api/v1/collectors/{id}. That still frees a seat; only un-decommissioning is now gated.
4. Collectors that were already deregistered before this upgrade are treated as decommissioned, because the reason was not recorded before now. If you intended to bring one back, do it while you have headroom.

WHAT HAPPENS IF YOU SKIP THIS. Nothing breaks at upgrade time. The failure arrives later and looks unrelated: someone presses Reset Connection on a collector and gets a refusal, or a collector that was deliberately stopped does not come back. Both are correct behaviour, and both are surprising if the count moved and nobody looked.

WHY THIS CHANGED. The count could previously be reset in a loop through documented endpoints — deregister to drop the count, enrol again, then reset the originals. Verified in a lab: ten collectors live under a limit of five.

Shipped in v1.342.8, 2026-08-31 — the release itself.

Security updates

No security updates in this release.

Fixes

107 fixes in this release have not been written up yet. They are not listed here because a description written for the people who built it is no use to somebody deciding whether to upgrade.

Features and improvements

23 changes in this release have not been written up yet. They are not listed here because a description written for the people who built it is no use to somebody deciding whether to upgrade.