Snippets · Sources

Scrape cAdvisor container metrics from the kubelet

Many Kubernetes dashboards are built on cAdvisor's container_* metrics. Each kubelet serves them at /metrics/cadvisor on port 10250; a DaemonSet collector scrapes its own node with the prometheus receiver, authenticated with the pod's service-account token. In our kind cluster it produced 88 metrics per scrape.

OpenTelemetry Collector

OpenTelemetry Collector YAML

receivers:
  prometheus/cadvisor:
    config:
      scrape_configs:
        - job_name: kubelet-cadvisor
          scrape_interval: 30s
          scheme: https
          metrics_path: /metrics/cadvisor
          authorization:
            credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          tls_config:
            insecure_skip_verify: true
          static_configs:
            - targets: ['${env:K8S_NODE_NAME}:10250']

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://your-backend:4318

service:
  pipelines:
    metrics:
      receivers: [prometheus/cadvisor]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

The collector needs this to run in a cluster: a ServiceAccount, the RBAC it uses and the workload. Create the config first with kubectl -n observability create configmap k8s-cadvisor-scrape --from-file=config.yaml.

Kubernetes manifest

apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-cadvisor-scrape
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-cadvisor-scrape
rules:
  - apiGroups: [""]
    resources: [nodes/metrics]
    verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-cadvisor-scrape
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k8s-cadvisor-scrape
subjects:
  - kind: ServiceAccount
    name: k8s-cadvisor-scrape
    namespace: observability
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: k8s-cadvisor-scrape
  namespace: observability
spec:
  selector:
    matchLabels:
      app: k8s-cadvisor-scrape
  template:
    metadata:
      labels:
        app: k8s-cadvisor-scrape
    spec:
      serviceAccountName: k8s-cadvisor-scrape
      containers:
        - name: otelcol
          image: otel/opentelemetry-collector-contrib:0.161.0
          args: ["--config=/conf/config.yaml"]
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: config
              mountPath: /conf
      volumes:
        - name: config
          configMap:
            name: k8s-cadvisor-scrape

LinkMesh

Library › Sources › Kubernetes Container Metrics (cAdvisor)

Scrape Targets: ["${env:K8S_NODE_NAME}:10250"]
Scheme: https
Scrape Interval: 30s
Metrics Path: /metrics/cadvisor
Bearer Token File: /var/run/secrets/kubernetes.io/serviceaccount/token
TLS: {"ca_file":"/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"}
Sample Limit: 50000
Metric Filters: [{"action":"drop","regex":"container_.*;","source_labels":["__name__","image"]},{"action":"drop","regex":"container_(?:[^n;]|n[^e;]|ne[^t;])[^;]*;","source_labels":["__name__","container"]}]

What LinkMesh then pushes to the collector, under receivers:

Pushed to the collector

prometheus/input_src_00000501:
  config:
    scrape_configs:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        job_name: input_src_00000501
        metric_relabel_configs:
          - action: drop
            regex: container_.*;
            source_labels:
              - __name__
              - image
          - action: drop
            regex: container_(?:[^n;]|n[^e;]|ne[^t;])[^;]*;
            source_labels:
              - __name__
              - container
          - action: keep
            regex: container_cpu_cfs_periods_total|container_cpu_cfs_throttled_periods_total|container_cpu_cfs_throttled_seconds_total|container_cpu_usage_seconds_total|container_fs_reads_bytes_total|container_fs_reads_total|container_fs_writes_bytes_total|container_fs_writes_total|container_memory_cache|container_memory_rss|container_memory_swap|container_memory_usage_bytes|container_memory_working_set_bytes|container_network_receive_bytes_total|container_network_receive_packets_dropped_total|container_network_receive_packets_total|container_network_transmit_bytes_total|container_network_transmit_packets_dropped_total|container_network_transmit_packets_total|machine_cpu_cores|machine_memory_bytes
            source_labels:
              - __name__
        metrics_path: /metrics/cadvisor
        sample_limit: 50000
        scheme: https
        scrape_interval: 30s
        static_configs:
          - targets:
              - ${env:K8S_NODE_NAME}:10250
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

The same, as an API call (see automation recipes):

API call

curl -X POST "$LINKMESH/sources" \
  -H "Authorization: Bearer $LMSAT" \
  -H "Content-Type: application/json" \
  -d '{
  "defaultConfig": {
    "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
    "metric_relabel_configs": [
      {
        "action": "drop",
        "regex": "container_.*;",
        "source_labels": [
          "__name__",
          "image"
        ]
      },
      {
        "action": "drop",
        "regex": "container_(?:[^n;]|n[^e;]|ne[^t;])[^;]*;",
        "source_labels": [
          "__name__",
          "container"
        ]
      }
    ],
    "metric_selection": {
      "catalog_source": "prometheus/cadvisor",
      "preset": "minimum"
    },
    "metrics_path": "/metrics/cadvisor",
    "sample_limit": 50000,
    "scheme": "https",
    "scrape_interval": "30s",
    "targets": [
      "${env:K8S_NODE_NAME}:10250"
    ],
    "tls_config": {
      "ca_file": "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
    }
  },
  "name": "Kubernetes Container Metrics (cAdvisor)",
  "receiverType": "prometheus"
}'

The rest of the collector config above (memory_limiter, batch, otlphttp) is not part of this template: in LinkMesh those are separate sources, processors and destinations on the route.

Generated from LinkMesh's template catalogue by linkmesh-server snippet-render 1.464.3, not written by hand.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0). Replace https://your-backend:4318 with your own OTLP endpoint before you run it.

The gotcha

The ClusterRole needs get on nodes/metrics, and without it the collector still looks healthy. In our test, with only nodes/stats in the role, the scrape failed with a warning, not an error, and the collector still emitted 5 metrics instead of 88 - the scrape bookkeeping series, no container data.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Doing this in LinkMesh: Install LinkMesh on Kubernetes

Install LinkMesh →See all snippets

Scrape cAdvisor container metrics from the kubelet

OpenTelemetry Collector

LinkMesh

The gotcha

Manage this across a fleet

Related snippets