Mask PII

Sensitive values — card numbers, email addresses, phone numbers, national IDs — slip into logs constantly. Once they reach an observability backend they are indexed, retained, and eventually queried by people who should never have seen them.

The PII Masking processor redacts them on the collector, before any data leaves the host. You build rules from a library of maintained detectors; LinkMesh compiles those rules into the collector’s own configuration and pushes it to the edge. There is no “trust the backend to scrub it later” step.

Masking works on both collector types

The masking processor is compiled by both config generators, so it runs on Grafana Alloy collectors (remotecfg) and on otelcol collectors (OpAMP) alike. Unlike live capture, it needs no co-located agent — masking is part of the pipeline the collector already runs.

How masking works

A masking rule is three choices: what to find (a detector, or your own pattern), what to do with it (a mode), and which text to look in (a scope).

What gets stored on the pipeline is those three choices — not the resulting expression. LinkMesh compiles the rules into collector configuration each time it generates a config. This matters more than it looks: a pipeline that stored a rendered expression would keep whatever the pattern looked like on the day it was saved, and would quietly stop matching new formats as the detector library improved. Storing the intent means an improved detector reaches every pipeline already using it.

A masking rule that cannot compile stops config delivery

If a rule set cannot be compiled, LinkMesh refuses to generate the collector’s config at all, and the collector keeps running the config it already has. That is deliberate. The alternative — dropping the processor and generating the rest — would send the pipeline’s records onward unmasked, and nothing would report it.

The cost is worth naming: one broken masking rule freezes config delivery for every route on that collector, not just its own. Rules are validated when you save them, so you should only ever meet this by hand-editing a config outside the UI.

1. Add the masking processor

Open the pipeline you want to redact in (or create one — see Build your first pipeline) and click + Add Processor, then choose PII Masking.

The processor picker. 'PII Masking' opens the rule builder; the generic 'Transform (custom)' entry is still there for non-PII rewriting.

2. Build a rule

1. Pick a detector. The picker groups the library by category — contact details, network addresses, financial identifiers, national IDs. Each detector is pinned to a version, so the rule you save keeps behaving the way it did when you tested it.

2. Pick a mode — what replaces the value it finds.

3. Pick a scope — which text the rule looks in.

4. Read the notes. The builder shows the detector’s own over-match and miss notes, plus what the chosen mode costs. These are the two reasons a masking rule behaves differently from what its name suggests, so they are shown before you save rather than discovered later in a dashboard.

5. Save. Modes that cannot work for the detector and scope you picked are never offered in the first place, so a rule you can build is a rule that compiles.

Screenshot pendingpipelines/pii-mask-rule-builder

The masking rule builder: detector, mode and scope per rule, with the detector's over-match and miss notes shown inline before saving.

The detector library

Detector	What it finds
email	Email address
phone_e164	Phone number (international)
phone_nanp	Phone number (North American)
ipv4	IPv4 address
ipv6	IPv6 address
card	Payment card number (shape only)
card_visa	Payment card — Visa
card_mastercard	Payment card — Mastercard
card_amex	Payment card — American Express
iban	IBAN (bank account number)
us_ssn	US Social Security Number
ch_ahv	Swiss AHV/AVS number
eu_vat	EU VAT identification number

Detectors are referenced with their version, as in card_visa@1. The builder always writes the pinned reference for you, so an improved detector never changes what a saved rule matches until you move it to the new version.

Every detector carries its own over-match and miss notes, shown in the builder before you save — read them, because they are the difference between what a detector is called and what it actually matches.

At the edge, detectors match on shape

Checksum validation — the Luhn check on a card number, mod-97 on an IBAN — runs on the server, in the dry-run preview only, to report how many matches are likely false positives. The collector masks on shape alone.

That is the safe direction: a value that looks like a card number gets masked whether or not it would pass a checksum. Expect some over-masking rather than under-masking, and use the dry-run to see how much.

Modes

Mode	What it does	What it costs
Full redaction	Replaces the detected value with a fixed token. Only the matched substring is replaced, so the rest of the line stays readable.	Every occurrence becomes the same token, so two lines about two different people are indistinguishable. Nothing masked this way can be correlated or counted per subject.
Consistent hash	Replaces the value with its SHA-256 digest. The same input always produces the same digest, so records about the same subject can still be joined without the value being present.	See the warning below — at body and all-attributes scope the hash is unsalted.
Partial reveal	Keeps a bounded, conventional part of the value — the last four digits of a card, the domain of an email — and masks the rest.	A revealed tail is still data leaving your pipeline. Not every detector has a reveal form; where none is defined the mode is unavailable rather than approximated.

Consistent hash is not anonymisation at substring scope

At body or all attributes scope the hash is unsalted — the matched substring is hashed and there is no way to fold a salt into it. For low-entropy data such as an email address, a phone number or a national ID, an unsalted hash can be reversed by hashing a dictionary of candidates.

Use it for correlation, not for protection. Where protection is the point, choose full redaction, partial reveal, or a salted hash of a named attribute — the one scope that can carry a salt.

The default replacement token is **** for every detector, deliberately: a token that named the detector would tell a reader of the log which class of sensitive data used to be there. You can override it per rule.

Scopes

Scope	What it replaces
Body	Matched substrings inside the log body.
All attributes	Matched substrings in every log attribute value.
Named attribute	The entire value of one attribute you name, and only when that whole value is the detected data. This is the only scope that can carry a salt.

The named-attribute scope matches the whole value rather than “contains”, because it is the only scope that can destroy surrounding text.

Rotating a salt breaks correlation across the rotation

Rotating the salt changes every hash from that moment on, so records masked before the rotation can no longer be correlated with records masked after it. The break is by design — it is what stops an old salt from remaining useful forever — but it is not reversible, and any saved query or dashboard that joins on a hashed value will silently stop matching across the boundary.

3. Mask something the library doesn’t cover

For values specific to your organisation — internal ticket IDs, employee numbers, customer references — choose Custom pattern in the detector picker and supply your own regular expression and a label.

A custom rule carries either a library detector or its own pattern, never both: two sources of truth for what a rule matches would mean nothing could report which one won.

Custom patterns use RE2 — the same engine the collector runs — so there are no backreferences and no lookaround. They offer full redaction and consistent hash only; partial reveal needs a per-detector notion of which part of a value is safe to show, and there is none for a pattern you just wrote. A pattern is also rejected if it can match empty (it would fire at every position and destroy the record) or if it is dramatically slower than the built-in detectors on a large line.

1. Enter the pattern and a label — the label is what identifies the rule in the list and in metrics.

2. Paste a sample line that contains the value you want masked.

3. Click Test. You see exactly what the rule would do to that line.

4. Save. For a new or changed pattern the server requires that the test matched. Read what that does and does not cover below.

The save gate is real on the API — and there are paths it does not cover

Since v1.339.2 a custom rule that is new, or whose pattern you changed, cannot be saved through the API unless the sample you tested it against is one the pattern actually matches. The rule carries that sample with it and the server re-runs the match on every save, so there is no separate test state that can go stale and no way to claim a test that never passed. Creating a pipeline enforces this on every custom rule in it.

Three cases are not covered:

• Rules you already have. A custom rule whose pattern, label, mode and scope are unchanged still saves without proof — deliberately, so that editing one part of a pipeline is never blocked by an untested rule somebody left in another part of it. The masking coverage view lists those rules as unverified so you can work through them on purpose rather than have an auditor find them.
• Rolling back. Restoring an earlier version does not require test proof, so a rollback can reinstate an untested rule. That is deliberate: a rollback is a fallback path, and demanding fresh proof would strand you on the very version you are trying to leave. The restored rules appear as unverified in the coverage view, exactly like any other untested rule. A rollback is checked for compilation, so a chain that could not compile is refused outright rather than reaching your collectors.
• Rule sets edited directly in Git. A masking config hand-edited in the deployment repository reaches config generation without going through the API, and so without this check. It still fails closed if it does not compile — but nothing proves it matches anything.

In every case the check proves only that your pattern fires against the sample you supplied — never that it matches the real values in your traffic. That distinction matters, because a masking rule that never fires is indistinguishable from a working one from every other vantage point: the pipeline looks configured, the generated config contains the rule, the dashboards show a masking processor in the chain, and nothing downstream reports that it matched nothing. Test with a value shaped like your production data, not the shortest string that satisfies the pattern.

4. Verify before you roll it out

Rules are validated on the server as you edit them, so the notes and the compiled result you read before saving come from the same code that will compile the rule for your fleet.

A salted hash rule cannot be previewed

Testing and dry-run both refuse a rule that uses a salted hash. The salt is a secret resolved only when the config is delivered to the collector, so any digest shown in a preview would be computed with the wrong salt and would not match what the collector actually produces. Showing a confident wrong answer is worse than refusing.

To check that the rule finds what you expect, test the same pattern and scope in full redaction mode, then switch back to the salted hash.

For an end-to-end check, open the dry-run panel and paste a sample event:

{
  "resourceLogs": [{
    "scopeLogs": [{
      "logRecords": [{
        "body": {
          "stringValue": "POST /api/v2/cart/checkout [email protected] card=4242 4242 4242 4242"
        }
      }]
    }]
  }]
}

Run it, and the Final Output panel shows the body rewritten with the matched values replaced.

Dry-run with masking applied: the body field is rewritten, with placeholders in place of the matched substrings.

5. Confirm it on a live collector

Save the pipeline and attach it via a route on a collector handling real traffic. The collector picks up the new config on its next fetch.

The fastest confirmation: add a debug exporter alongside your real destination for a minute, watch journalctl -u alloy -f (or -u otelcol-contrib -f), and confirm that outbound records carry the masked values rather than the raw ones. Remove the debug exporter once you have seen one masked record — it is noisy at sustained traffic.

Tip

Capture live samples is the less noisy way to do the same check on an otelcol collector with a co-located agent: tap the route post-process and read the records that are actually leaving.

Related

• Pipeline — the reusable processor chain this recipe modifies
• Route — wires the pipeline to a source-destination pair on a collector
• Drop noisy logs — the Filter sibling of this recipe
• Capture live samples — read the records actually leaving a route
• Security & encryption — how secrets such as a masking salt are stored and delivered
• Trust on linkmesh.io — why we redact at the collector edge rather than the backend
• Masking PII in logs — a worked walkthrough of the same approach