LinkMesh

Search docs, blog and changelog

ENDE

Snippets · Sources

Collect Kubernetes pod logs (DaemonSet)

Container runtimes write each container's output to /var/log/pods on the node. A DaemonSet collector mounts that directory and tails it; the container operator parses the containerd, CRI-O and Docker formats and reads k8s.namespace.name, k8s.pod.name, k8s.pod.uid and k8s.container.name from the file path, so no per-runtime regex is needed. In our kind cluster it shipped 611 records with all four attributes. The DaemonSet mounts /var/log/pods read-only from the host.

Signals
Logs
Category
CollectionKubernetes
Source
Kubernetes pod logs (/var/log/pods)
Components
filelogbatchmemory_limiterotlphttp
Destination
OTLP/HTTP endpoint

OpenTelemetry Collector

OpenTelemetry Collector YAML
extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true

receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    # The collector's own logs would otherwise be read back and re-shipped.
    exclude: [/var/log/pods/observability_k8s-pod-logs*/*/*.log]
    start_at: end
    include_file_path: true
    storage: file_storage
    operators:
      # Parses the CRI-O, containerd and Docker log formats and reads
      # namespace, pod, uid and container from the file path.
      - type: container
        id: container-parser

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://your-backend:4318

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [filelog]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

The collector needs this to run in a cluster: a ServiceAccount, the RBAC it uses and the workload. Create the config first with kubectl -n observability create configmap k8s-pod-logs --from-file=config.yaml.

Kubernetes manifest
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-pod-logs
  namespace: observability
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: k8s-pod-logs
  namespace: observability
spec:
  selector:
    matchLabels:
      app: k8s-pod-logs
  template:
    metadata:
      labels:
        app: k8s-pod-logs
    spec:
      serviceAccountName: k8s-pod-logs
      securityContext:
        # /var/log/pods is readable by root only on most nodes.
        runAsUser: 0
        runAsGroup: 0
      containers:
        - name: otelcol
          image: otel/opentelemetry-collector-contrib:0.161.0
          args: ["--config=/conf/config.yaml"]
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: config
              mountPath: /conf
            - name: varlogpods
              mountPath: /var/log/pods
              readOnly: true
            - name: storage
              mountPath: /var/lib/otelcol/storage
      volumes:
        - name: config
          configMap:
            name: k8s-pod-logs
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: storage
          hostPath:
            path: /var/lib/otelcol/k8s-pod-logs
            type: DirectoryOrCreate

LinkMesh

Library › Sources › Kubernetes Pod Logs

Include Namespaces
[]
Exclude Namespaces
[]
Include Workloads
[]
Exclude Workloads
[]
Resume After Restart
true
Start Reading At
end

What LinkMesh then pushes to the collector, under receivers:

Pushed to the collector
filelog/input_src_00000501:
  include:
    - /var/log/pods/*/*/*.log
  include_file_path: true
  operators:
    - type: container
  start_at: end
  storage: file_storage/input_src_00000501

# extensions:
file_storage/input_src_00000501:
  create_directory: true
  directory: /var/lib/otelcol-supervisor/filelog-storage/input_src_00000501

The same, as an API call (see automation recipes):

API call
curl -X POST "$LINKMESH/sources" \
  -H "Authorization: Bearer $LMSAT" \
  -H "Content-Type: application/json" \
  -d '{
  "defaultConfig": {
    "checkpoint": true,
    "namespace_exclude": [],
    "namespace_include": [],
    "start_at": "end",
    "workload_exclude": [],
    "workload_include": []
  },
  "name": "Kubernetes Pod Logs",
  "receiverType": "k8s_pod_logs"
}'

The rest of the collector config above (memory_limiter, batch, otlphttp) is not part of this template: in LinkMesh those are separate sources, processors and destinations on the route.

Generated from LinkMesh's template catalogue by linkmesh-server snippet-render 1.464.3, not written by hand.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0). Replace https://your-backend:4318 with your own OTLP endpoint before you run it.

The gotcha

The collector image runs as uid 10001, and on the node the log files are root-owned: /var/log/pods was root:root 0750 and the files 0640 on our containerd node. Run as the default user and it does not even get that far - in our test it crash-looped on "storage client: open /var/lib/otelcol/storage/receiver_filelog_: permission denied" for the root-owned hostPath checkpoint directory. The manifest runs the container as root for that reason.

Used in these pipelines

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Doing this in LinkMesh: Install LinkMesh on Kubernetes

Install LinkMesh →See all snippets

Related snippets