Snippets · Sources

Scrape pods by prometheus.io annotations

Teams opt their pods into scraping with prometheus.io/scrape, /port and /path annotations. The prometheus receiver discovers pods through the Kubernetes API, keeps the annotated ones and rewrites the target from the annotations; the field selector makes each DaemonSet pod discover only the pods on its own node. In our kind cluster it scraped an annotated collector pod and labelled its metrics with k8s_namespace_name and k8s_pod_name.

OpenTelemetry Collector

OpenTelemetry Collector YAML

receivers:
  prometheus/annotated-pods:
    config:
      scrape_configs:
        - job_name: annotated-pods
          scrape_interval: 30s
          kubernetes_sd_configs:
            - role: pod
              selectors:
                # Each DaemonSet pod discovers only the pods on its own node.
                - role: pod
                  field: spec.nodeName=${env:K8S_NODE_NAME}
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: "true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              action: replace
              target_label: __metrics_path__
              regex: (.+)
            - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $$1:$$2
              target_label: __address__
            - source_labels: [__meta_kubernetes_namespace]
              target_label: k8s_namespace_name
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: k8s_pod_name

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://your-backend:4318

service:
  pipelines:
    metrics:
      receivers: [prometheus/annotated-pods]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

The collector needs this to run in a cluster: a ServiceAccount, the RBAC it uses and the workload. Create the config first with kubectl -n observability create configmap k8s-pod-autodiscovery --from-file=config.yaml.

Kubernetes manifest

apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-pod-autodiscovery
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-pod-autodiscovery
rules:
  - apiGroups: [""]
    resources: [pods]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-pod-autodiscovery
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k8s-pod-autodiscovery
subjects:
  - kind: ServiceAccount
    name: k8s-pod-autodiscovery
    namespace: observability
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: k8s-pod-autodiscovery
  namespace: observability
spec:
  selector:
    matchLabels:
      app: k8s-pod-autodiscovery
  template:
    metadata:
      labels:
        app: k8s-pod-autodiscovery
    spec:
      serviceAccountName: k8s-pod-autodiscovery
      containers:
        - name: otelcol
          image: otel/opentelemetry-collector-contrib:0.161.0
          args: ["--config=/conf/config.yaml"]
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: config
              mountPath: /conf
      volumes:
        - name: config
          configMap:
            name: k8s-pod-autodiscovery

LinkMesh

Library › Sources › Kubernetes Pod Autodiscovery (prometheus.io annotations)

Kubernetes Discovery: {"role":"pod"}
Scheme: http
Scrape Interval: 30s
Metrics Path: /metrics
Sample Limit: 10000

What LinkMesh then pushes to the collector, under receivers:

Pushed to the collector

prometheus/input_src_00000501:
  config:
    scrape_configs:
      - job_name: input_src_00000501
        kubernetes_sd_configs:
          - role: pod
            selectors:
              - field: spec.nodeName=${env:K8S_NODE_NAME}
                role: pod
        metrics_path: /metrics
        relabel_configs:
          - action: keep
            regex: "true"
            source_labels:
              - __meta_kubernetes_pod_annotation_prometheus_io_scrape
          - action: drop
            regex: Pending|Succeeded|Failed|Unknown
            source_labels:
              - __meta_kubernetes_pod_phase
          - action: drop
            regex: "true"
            source_labels:
              - __meta_kubernetes_pod_container_init
          - regex: (.+:[0-9]+)
            separator: ':'
            source_labels:
              - __meta_kubernetes_pod_ip
              - __meta_kubernetes_pod_annotation_prometheus_io_port
            target_label: __address__
          - regex: (.+)
            source_labels:
              - __meta_kubernetes_pod_annotation_prometheus_io_path
            target_label: __metrics_path__
          - regex: (https?)
            source_labels:
              - __meta_kubernetes_pod_annotation_prometheus_io_scheme
            target_label: __scheme__
          - regex: (.+)
            source_labels:
              - __meta_kubernetes_pod_uid
            target_label: linkmesh_k8s_pod_uid
          - regex: (.*)
            replacement: kubernetes-pods
            target_label: job
        sample_limit: 10000
        scheme: http
        scrape_interval: 30s

The same, as an API call (see automation recipes):

API call

curl -X POST "$LINKMESH/sources" \
  -H "Authorization: Bearer $LMSAT" \
  -H "Content-Type: application/json" \
  -d '{
  "defaultConfig": {
    "kubernetes_sd": {
      "role": "pod"
    },
    "metrics_path": "/metrics",
    "sample_limit": 10000,
    "scheme": "http",
    "scrape_interval": "30s"
  },
  "name": "Kubernetes Pod Autodiscovery (prometheus.io annotations)",
  "receiverType": "prometheus"
}'

The rest of the collector config above (memory_limiter, batch, otlphttp) is not part of this template: in LinkMesh those are separate sources, processors and destinations on the route.

Generated from LinkMesh's template catalogue by linkmesh-server snippet-render 1.464.3, not written by hand.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0). Replace https://your-backend:4318 with your own OTLP endpoint before you run it.

The gotcha

prometheus.io/scrape alone is not enough. In our test a pod annotated scrape: "true" without a prometheus.io/port annotation, and declaring no container port, was scraped at its pod IP on port 80: "dial tcp 10.244.0.23:80: connect: connection refused", a warning in the collector log and no metrics. Set prometheus.io/port, or declare the metrics port on the container.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Doing this in LinkMesh: Install LinkMesh on Kubernetes

Install LinkMesh →See all snippets

Scrape pods by prometheus.io annotations

OpenTelemetry Collector

LinkMesh

The gotcha

Manage this across a fleet

Related snippets