Regulated Kubernetes

Kubernetes observability is a solved problem in the tutorial sense. Install a DaemonSet, scrape kubelet, tail /var/log/pods, send it somewhere. Every conference talk covers it and every managed platform ships a version of it. What the tutorials do not cover is the review that follows in a bank, an insurer or a hospital: which of those pod logs contain client data, who can read them, where did they go, and can you prove what the collector on node 14 was configured to do on the day of the incident?

Those questions are not harder on Kubernetes than on VMs. They are harder to notice, because the platform’s abstractions — namespaces, ephemeral pods, an in-cluster monitoring stack that comes for free — hide the data-handling decisions that a regulator will later ask about. This guide is about surfacing them, on vanilla Kubernetes and on OpenShift, which adds a few of its own.

Five things the tutorial setup gets wrong for regulated workloads

• Namespaces are tenants, and the default collector does not know it. A DaemonSet-per-node collector sees every pod on the node regardless of namespace. Unless the pipeline stamps the namespace as a resource attribute and routes on it, the trading desk’s application logs and the HR system’s logs leave the node in one stream to one destination. That is a data-segregation finding waiting to happen.
• Pod logs are where the PII is. Application logs on Kubernetes are stdout, and stdout is whatever the developer printed — request bodies, user objects, tokens in URLs. The masking has to happen in the collector on the node, before the record leaves it, because nothing upstream of that point sees the content — masking PII in logs.
• The bundled monitoring stack is not the audit trail. The Prometheus and Alertmanager that ship with the platform monitor the platform. They are not designed to carry, retain or govern application telemetry under a retention obligation, and treating them as the system of record is how a review discovers that the evidence expired after fifteen days.
• The API server audit log is evidence, and it is usually not shipped. Who deleted the deployment, who read the secret, who changed the RBAC binding — that is the kube-apiserver audit log, and in most clusters it goes to a file on the control plane that nobody collects. In a regulated estate it is one of the first sources to onboard, and it belongs in a different destination from application logs.
• Egress is a control, not a config detail. A collector that exports to a SaaS backend is an egress path out of the cluster. It should be a deliberate one — a known gateway, a NetworkPolicy that allows only that path, credentials held centrally — not four DaemonSets each with their own outbound connection to a different vendor.

What OpenShift adds

OpenShift is Kubernetes with opinions, and three of them matter here:

• Security Context Constraints. A collector that tails /var/log/pods needs a hostPath mount and a privileged-ish SCC. OpenShift will not grant that by default, which is correct — and it means the collector’s permissions are an explicit, reviewable object rather than an implicit default. Treat the SCC as part of the evidence.
• Its own logging and tracing operators. OpenShift ships a logging stack and a build of the OpenTelemetry Collector as operators. Both are legitimate choices. The question to answer up front is whether the collection layer is managed by the platform operator (cluster-scoped, Red Hat’s lifecycle) or by your fleet control plane (the same config as your VMs and other clusters). Mixing the two on one cluster produces exactly the “which agent owns this file” confusion covered in one strategy, multiple backends.
• Projects as a hard tenancy model. OpenShift projects are namespaces with an admission layer around them. If you route telemetry by project, the platform’s own tenancy boundary and the observability tenancy boundary line up, which is the property auditors like most.

A pipeline shape that passes review

Concretely, the collector configuration on each node does four things the tutorial version does not:

processors:
  # 1. Stamp the tenant. Namespace becomes a resource attribute on every record.
  k8sattributes:
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name]
  # 2. Mask on the node, before egress.
  transform/redact:
    log_statements:
      - context: log
        statements:
          - replace_pattern(body, "[\\w.+-]+@[\\w-]+\\.[\\w.]+", "<email>")
          - replace_pattern(body, "(?i)(bearer\\s+)[a-z0-9._-]+", "$$1<token>")
  batch:

connectors:
  # 3. Route by tenant. Regulated namespaces stay on-prem; the rest may leave.
  routing:
    default_pipelines: [logs/onprem]
    table:
      - context: resource
        condition: attributes["k8s.namespace.name"] == "trading"
        pipelines: [logs/onprem]
      - context: resource
        condition: attributes["k8s.namespace.name"] == "platform"
        pipelines: [logs/cloud]

The fourth thing is not in the YAML: the YAML comes from somewhere authoritative. A node running last month’s config — the one without the masking rule — is not a tidiness problem in a regulated cluster; it is an unredacted-data incident. Central, versioned config pushed to every node, with a record of who changed it, is what turns “we mask PII” from a statement into a control — detecting config drift.

The sources to onboard, in order

1. Application logs from the regulated namespaces, masked on the node, routed on-prem. This is where the exposure is.
2. The API server audit log, to a destination with the retention your regime requires and access limited to the people who investigate.
3. Kubelet and node metrics, which are low-sensitivity and can go wherever is cheapest.
4. Platform components last — they are already covered by the bundled stack, and duplicating them into your pipeline is cost without evidence value.

Honest limits

• Ephemeral pods make “what was running” harder. A pod that lived four minutes and was replaced does not leave a host to inspect. The record of what it emitted, and what the collector did to that, is the evidence — which is one more reason the collector config must be authoritative and auditable.
• Multi-cluster multiplies everything. Three clusters are three collector fleets with three drift surfaces. The answer is the same one as for VMs: one control plane, one config per role, rolled out everywhere — fleet management.
• Service meshes and sidecars change the picture. If you run a mesh, a share of the telemetry — and the mTLS boundary — moves into the sidecar. That is a separate design question and this post does not pretend to cover it.