Snippets · Pipelines

Kubernetes pod logs to Loki, enriched with deployment and cluster

A DaemonSet collector tails /var/log/pods, the container operator reads namespace, pod and container from the path, k8sattributes adds the deployment, node and pod uid from the Kubernetes API, a transform stamps the cluster name, and otlphttp ships it to Loki's native OTLP endpoint. In our kind cluster it shipped 1,153 records, each with k8s.deployment.name (coredns, local-path-provisioner) and k8s.cluster.name set.

Included steps

OpenTelemetry Collector

OpenTelemetry Collector YAML

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true

receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    # The collector's own logs would otherwise be read back and re-shipped.
    exclude: [/var/log/pods/observability_k8s-logs-to-loki*/*/*.log]
    start_at: end
    include_file_path: true
    storage: file_storage
    operators:
      # Parses the CRI-O, containerd and Docker log formats and reads
      # namespace, pod, uid and container from the file path.
      - type: container
        id: container-parser

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # Adds deployment, node and pod uid from the Kubernetes API, matched by
  # the pod name and namespace the container operator read from the path.
  k8sattributes:
    auth_type: serviceAccount
    filter:
      node_from_env_var: K8S_NODE_NAME
    extract:
      metadata: [k8s.deployment.name, k8s.node.name, k8s.pod.uid]
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.name
          - from: resource_attribute
            name: k8s.namespace.name
  transform/cluster:
    error_mode: ignore
    log_statements:
      - context: resource
        statements:
          - set(attributes["k8s.cluster.name"], "${env:CLUSTER_NAME}")
  batch:
    timeout: 5s

exporters:
  otlphttp/loki:
    endpoint: http://loki.loki.svc.cluster.local:3100/otlp
    headers:
      X-Scope-OrgID: ${env:LOKI_TENANT}
    sending_queue:
      enabled: true
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [filelog]
      processors: [memory_limiter, k8sattributes, transform/cluster, batch]
      exporters: [otlphttp/loki]

The collector needs this to run in a cluster: a ServiceAccount, the RBAC it uses and the workload. Create the config first with kubectl -n observability create configmap k8s-logs-to-loki --from-file=config.yaml.

Kubernetes manifest

apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-logs-to-loki
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-logs-to-loki
rules:
  - apiGroups: [""]
    resources: [pods, namespaces, nodes]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [replicasets]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-logs-to-loki
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k8s-logs-to-loki
subjects:
  - kind: ServiceAccount
    name: k8s-logs-to-loki
    namespace: observability
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: k8s-logs-to-loki
  namespace: observability
spec:
  selector:
    matchLabels:
      app: k8s-logs-to-loki
  template:
    metadata:
      labels:
        app: k8s-logs-to-loki
    spec:
      serviceAccountName: k8s-logs-to-loki
      securityContext:
        # /var/log/pods is readable by root only on most nodes.
        runAsUser: 0
        runAsGroup: 0
      containers:
        - name: otelcol
          image: otel/opentelemetry-collector-contrib:0.161.0
          args: ["--config=/conf/config.yaml"]
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CLUSTER_NAME
              value: prod-eu-1
            - name: LOKI_TENANT
              valueFrom:
                secretKeyRef:
                  name: loki-tenant
                  key: tenant
          volumeMounts:
            - name: config
              mountPath: /conf
            - name: varlogpods
              mountPath: /var/log/pods
              readOnly: true
            - name: storage
              mountPath: /var/lib/otelcol/storage
      volumes:
        - name: config
          configMap:
            name: k8s-logs-to-loki
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: storage
          hostPath:
            path: /var/lib/otelcol/k8s-logs-to-loki
            type: DirectoryOrCreate

LinkMesh

Library › Sources › Kubernetes Pod Logs

Include Namespaces: []
Exclude Namespaces: []
Include Workloads: []
Exclude Workloads: []
Resume After Restart: true
Start Reading At: end

What LinkMesh then pushes to the collector, under receivers:

Pushed to the collector

filelog/input_src_00000501:
  include:
    - /var/log/pods/*/*/*.log
  include_file_path: true
  operators:
    - type: container
  start_at: end
  storage: file_storage/input_src_00000501

# extensions:
file_storage/input_src_00000501:
  create_directory: true
  directory: /var/lib/otelcol-supervisor/filelog-storage/input_src_00000501

The same, as an API call (see automation recipes):

API call

curl -X POST "$LINKMESH/sources" \
  -H "Authorization: Bearer $LMSAT" \
  -H "Content-Type: application/json" \
  -d '{
  "defaultConfig": {
    "checkpoint": true,
    "namespace_exclude": [],
    "namespace_include": [],
    "start_at": "end",
    "workload_exclude": [],
    "workload_include": []
  },
  "name": "Kubernetes Pod Logs",
  "receiverType": "k8s_pod_logs"
}'

The rest of the collector config above (memory_limiter, k8sattributes, transform, batch, otlphttp) is not part of this template: in LinkMesh those are separate sources, processors and destinations on the route.

Generated from LinkMesh's template catalogue by linkmesh-server snippet-render 1.464.3, not written by hand.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0).

The gotcha

The deployment name needs replicasets in the ClusterRole, and its absence is silent. In our test, with replicasets removed from the role, no record carried k8s.deployment.name and the collector logged no error or warning - logs simply arrived without it. The manifest grants get/list/watch on replicasets for that reason.

What it does not catch

No before/after table: k8sattributes needs a live Kubernetes API, and CI runs the collector outside a cluster. The figures above are from a measured run in a kind cluster.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Install LinkMesh →See all snippets