Snippets · Sources

Tail a log file

The oldest job in observability: an application writes to a file, you need it in your backend, and you need that to survive a collector restart without duplicating or losing lines. The example parses JSON lines with a ts field; change include and the parser to match your files.

OpenTelemetry Collector YAML

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true

receivers:
  filelog:
    include: [/var/log/myapp/*.log]
    start_at: end
    storage: file_storage
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.ts
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
  batch: {}

exporters:
  otlphttp:
    endpoint: https://your-backend:4318

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [filelog]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

Grafana Alloy

// stability: public-preview — start Alloy with --stability.level=public-preview,
// or it exits at startup instead of running without the component.

otelcol.storage.file "logs" {
	directory        = "/var/lib/otelcol/storage"
	create_directory = true
}

otelcol.receiver.filelog "app" {
	include  = ["/var/log/myapp/*.log"]
	start_at = "end"
	storage  = otelcol.storage.file.logs.handler

	operators = [
		{
			type      = "json_parser",
			timestamp = {
				parse_from = "attributes.ts",
				layout     = "%Y-%m-%dT%H:%M:%S.%LZ",
			},
		},
	]

	output {
		logs = [otelcol.processor.memory_limiter.default.input]
	}
}

otelcol.processor.memory_limiter "default" {
	check_interval   = "1s"
	limit_percentage = 80

	output {
		logs = [otelcol.processor.batch.default.input]
	}
}

otelcol.processor.batch "default" {
	output {
		logs = [otelcol.exporter.otlphttp.default.input]
	}
}

otelcol.exporter.otlphttp "default" {
	client {
		endpoint = "https://your-backend:4318"
	}
}

Alloy runs components below generally-available only when you ask it to, so start it with --stability.level=public-preview. Without the flag Alloy exits at startup instead of running with the component disabled.

Complete config files, each checked with the tool's own validate command (otelcol-contrib v0.161.0, Grafana Alloy v1.19.2). Replace https://your-backend:4318 with your own OTLP endpoint before you run it.

The gotcha

Without a file_storage extension the collector forgets its read position and starts over from start_at after every restart. And file_storage refuses to start when its directory does not exist, which is why create_directory: true is in this config.

Skip the YAML on every host

LinkMesh has this job built in as the File Tail source template: activate it on a collector from the control plane instead of editing config files host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Install LinkMesh →See all snippets

Tail a log file

The gotcha

Skip the YAML on every host

Related snippets