One Strategy, Many Backends

Most enterprises of any size run three or four observability backends at once. Splunk because the SOC standardised on it a decade ago. Sentinel because it came with E5 and security wanted Microsoft-native. Grafana because the platform team likes it and it is cheap. Dynatrace because one business-critical application was onboarded during a transformation programme and nobody is touching it.

This is usually described as sprawl, and the recommended fix is consolidation. In practice consolidation projects mostly fail, because each of those four has a real owner with a real reason, and “use my tool instead” is not an argument that wins against a compliance obligation or a renewal that runs to 2029.

The pragmatic position is that multiple backends are permanent, and the problem to solve is not how many you have — it is how many times you collect the same data.

The actual problem: N agents per host

Walk onto a host in an estate like this and count the agents. A Splunk universal forwarder, the Azure Monitor Agent, a Prometheus node exporter, a Dynatrace OneAgent. Four processes, four config mechanisms, four upgrade cycles, four sets of credentials — all reading substantially the same files and the same /proc.

The costs of that are concrete:

• You pay for the same data more than once. The same log line is collected, transmitted and ingested by two or three vendors independently.
• Hosts get double-counted. Several platforms bill per monitored host. Two agents reporting the same machine can produce two billable hosts, which is a pure-waste line item that is easy to miss.
• Resource footprint multiplies on machines that were sized for the workload, not for the monitoring.
• Nothing is consistent. Each agent has its own idea of what a host is called, so correlating across backends means reconciling three naming schemes.
• Every backend change is a fleet change, because collection is welded to destination.

Collect once, process once, fan out

The alternative is a single collection path per host, and fan-out at the end of it. One OpenTelemetry Collector reads the files and the host metrics; a single pipeline does the shared work — parsing, enrichment, redaction; and multiple exporters send to multiple destinations.

service:
  pipelines:
    logs:
      receivers: [filelog, journald, otlp]
      processors: [resourcedetection, transform/redact, batch]
      exporters: [splunk_hec, otlp/grafana, otlphttp/archive]

Three properties matter here and are worth stating explicitly:

• One read, one parse, one redaction. The expensive and security-relevant work happens once, which means a masking rule cannot be correctly applied on the path to one backend and forgotten on the path to another.
• Destinations are independent. Adding, removing or replacing a backend is an exporter change, not a fleet-wide agent operation.
• Fan-out is not duplication of effort, but it is duplication of data — and that has a cost and a correctness question, covered next.

The four traps

Fan-out has predictable failure modes. All four are avoidable and all four are common:

• Running two collection agents for the same source. The classic mistake during a migration: leave the old vendor agent tailing the file and add a collector tailing the same file. Now the host is double-counted and every log line exists twice downstream. Move the collection, then fan out from one place — dual-shipping without duplicates.
• Sending everything everywhere. Fan-out does not mean every destination gets the full stream. Security records belong in the SIEM; verbose application debug logs do not, and putting them there is how a Sentinel bill doubles. Split by attribute and send each destination what it is for — routing by attribute.
• Ignoring per-destination requirements. Backends want different things: a Splunk index and sourcetype, Sentinel’s table shape, a Grafana Cloud tenant label. That is per-destination processing on a branch, not a reason to run separate collection.
• One credential set for everything. Each destination needs its own credential, and rotating one should not require touching every host — which is an argument for holding destination credentials centrally rather than inline in per-node YAML (managing secrets).

Governance follows fan-out

Once one pipeline feeds four backends owned by four teams, some questions become organisational rather than technical:

• Who is allowed to add a destination? An exporter to a new SaaS endpoint is a data egress decision, and it should not be something any engineer can add to a config unseen.
• Who owns a route’s volume? If the SOC’s route triples in volume, whose budget is that and who is told? Per-route throughput makes this answerable — seeing real throughput.
• What stops one destination’s problem becoming everyone’s? A backend that stops accepting data creates backpressure. Queue behaviour and retry limits per exporter decide whether that becomes a local problem or a pipeline stall — collector high availability.

Where the four backends serve four different tenants or business units rather than four functions, this becomes the full multi-tenant question — multi-tenant collector architecture.

The strategic benefit nobody plans for

The reason to do this is efficiency. The reason it turns out to matter is optionality.

Once collection is decoupled from destination, adding a fifth backend is a configuration change, and — more importantly — removing one is too. The evaluation you could never justify becomes a route to a trial endpoint for two weeks. The renewal negotiation changes character entirely when the honest answer to “what would it take to leave” is “an exporter block and a fortnight” rather than “eighteen months and every host”.

That is the same argument as the end of vendor lock-in, arriving through the back door of a cost-reduction project.

Where to start

1. Count the agents on one representative host. This is usually the slide that gets the project approved.
2. Pick the least contentious backend — usually the platform team’s own — and move its collection to a collector, leaving the others untouched.
3. Add a second destination to that same pipeline rather than a second agent. Verify both backends see matching volume before removing anything.
4. Then retire agents one at a time, per source, keeping the collection layer constant.
5. Split the stream by value once fan-out works, so each backend receives what it is for instead of everything.