The Missing Layer

Draw the architecture of almost any observability stack and you get two boxes: things that produce telemetry, and a platform that stores and queries it. An agent connects them. That is the whole picture, and it has been the whole picture for twenty years.

Now draw the architecture of any other data domain in your organisation. Transactional data does not go straight from application to warehouse — there is ingestion, validation, transformation, routing. Message-driven systems have brokers. Even the humble web tier has a reverse proxy in front of it, because nobody wants clients talking directly to application servers.

Observability skipped that step. Not for a good architectural reason — for a commercial one. This post is about why the layer is missing, what its absence costs, and why it is arriving now whether or not anyone planned for it.

If you want the mechanics — what a pipeline is made of and how to run one — start with what is a telemetry pipeline. This post is the argument for why the layer exists at all.

Why the layer never formed

The agent came from the backend vendor. That single fact explains most of it.

If you bought Splunk, you got forwarders. Datadog gave you the Datadog Agent. New Relic gave you its APM agent, Elastic gave you Beats. In each case the collection layer was built, given away, and maintained by whoever was selling the storage — and it was optimised for that vendor’s ingest, priced into that vendor’s contract, and configured through that vendor’s console.

That arrangement is very convenient and quietly consequential. It means:

• Collection was never a design decision. You did not choose an agent; you inherited one when you chose a backend.
• Nobody was incentivised to send less. The organisation building your collection layer bills by the gigabyte it receives. There is no conspiracy here — just an absence of pressure in the direction of reduction.
• The layer had no independent existence, so no standards formed around it, so every vendor’s agent stayed incompatible with every other, so switching backends meant touching every host.

Compare the database world, which went through the same thing and came out differently. Applications used to embed vendor-specific client logic and talk straight to one database. The middle tier formed because the coupling became intolerable at scale — and once it existed, it became the place caching, pooling, routing and access control lived.

Observability is at the point where the coupling has become intolerable, roughly twenty years later.

What the missing layer costs

The absence is not abstract. It shows up as four recurring, expensive symptoms:

• Cost is decided by whoever produces the logs. Volume is set at the edge by a developer’s log statement and discovered at the end of the month on an invoice. Nothing in between arbitrates. Backend-side controls — retention tiers, sampling, index policies — all operate on data you have already transmitted and already paid to ingest.
• Compliance is retroactive. Whether client data left the network was decided by whatever the agent was configured to collect, months before anyone asked the question. The controls available afterwards are access controls on a copy that already crossed the boundary.
• Backend migration means re-instrumentation. The most expensive property of the two-box architecture. Changing where telemetry is stored should be a routing change; in the absence of a middle tier it is a fleet-wide agent replacement.
• There is no place to put cross-cutting logic. Every organisation eventually wants the same things — mask this field everywhere, drop health checks everywhere, tag every record with the owning team. With no shared layer, each of those becomes N implementations in N services, and one of them is always wrong.

Each symptom gets treated locally: a cost project, a compliance review, a migration project. They are the same structural gap presenting four times.

What the layer actually is

A telemetry pipeline is a tier that sits between producers and backends and owns four decisions:

• What is collected, uniformly, without every team reinventing it.
• What it contains when it leaves — redaction and enrichment applied as policy rather than as a code review comment.
• How much of it there is — filtering, sampling and aggregation applied before egress, where they are cheap, rather than after ingest, where they are not.
• Where it goes — per record, by attribute, to one or several destinations.

The reason OpenTelemetry matters to this story is not the SDK. It is that OTLP and the Collector made the middle tier possible to build without a vendor’s permission — an open wire format and an open, programmable component that sits in the path. The standard did not create demand for the layer; the demand was already there. It removed the thing that prevented it forming.

The objection worth taking seriously

“This is just another component to run, and it sits in the critical path of my observability data.”

That is correct, and it is the honest cost. A pipeline tier can fail, needs its own availability design, and adds a place where a bad config drops records you needed — which is why previewing changes against real data and gating rollouts matters more here than in most infrastructure. Collector high availability covers what running it properly involves.

The counter-argument is not that the cost is small. It is that you are already paying a larger version of it, distributed and invisible: the coupling tax on every migration, the compliance exposure you cannot see, and the volume nobody arbitrates. A middle tier makes that cost explicit and central, which is what makes it manageable.

What to take from this

If your architecture diagram has two boxes and an arrow, the layer is not absent — it is implicit, distributed across every agent config on every host, owned by nobody, and optimised by your backend vendor.

Making it explicit does not require a big-bang project. It requires putting one component in the path that you control, and then moving decisions into it one at a time: first volume, then redaction, then routing. Every one of those is independently valuable, and together they are the tier.