LinkMesh

Search docs, blog and changelog

ENDE

Snippets · Sources

Parse Apache access logs into semantic conventions

Apache httpd's combined log format carries client, user, time, request line, status, bytes, referer and user agent in one line. This config parses it on the filelog receiver and maps each field to its OpenTelemetry semantic-convention name, with the status code and body size as integers and the record timestamp taken from the line.

Signals
Logs
Category
CollectionParsing
Source
Apache access log
Components
filelogbatchmemory_limitertransformotlphttp
Destination
OTLP/HTTP endpoint

OpenTelemetry Collector

OpenTelemetry Collector YAML
extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true

receivers:
  filelog:
    include: [/var/log/apache2/access.log]
    start_at: end
    storage: file_storage
    operators:
      # Combined log format: client, user, time, request line, status, bytes,
      # referer, user agent. The request line is split into method, path,
      # query and protocol version.
      - type: regex_parser
        regex: '^(?P<client_address>\S+) \S+ (?P<user_name>\S+) \[(?P<time>[^\]]+)\] "(?P<method>[A-Z]+) (?P<path>[^ ?"]*)(?:\?(?P<query>[^ "]*))? HTTP/(?P<protocol_version>[0-9.]+)" (?P<status>\d{3}) (?P<size>\d+|-) "(?P<referer>[^"]*)" "(?P<user_agent>[^"]*)"'
        timestamp:
          parse_from: attributes.time
          layout: '%d/%b/%Y:%H:%M:%S %z'
        on_error: send

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # Maps the parsed fields to OpenTelemetry semantic convention names and
  # types (status code and body size as integers).
  transform/semconv:
    error_mode: ignore
    log_statements:
      - context: log
        # Only lines the regex parsed: an unparsed line keeps its body and
        # gets none of these attributes.
        conditions:
          - attributes["method"] != nil
        statements:
          - set(attributes["client.address"], attributes["client_address"])
          - set(attributes["user.name"], attributes["user_name"]) where attributes["user_name"] != "-"
          - set(attributes["http.request.method"], attributes["method"])
          - set(attributes["url.path"], attributes["path"])
          - set(attributes["url.query"], attributes["query"]) where attributes["query"] != nil and attributes["query"] != ""
          - set(attributes["network.protocol.name"], "http")
          - set(attributes["network.protocol.version"], attributes["protocol_version"])
          - set(attributes["http.response.status_code"], Int(attributes["status"]))
          - set(attributes["http.response.body.size"], Int(attributes["size"])) where attributes["size"] != "-"
          - set(attributes["http.request.header.referer"], [attributes["referer"]]) where attributes["referer"] != "-"
          - set(attributes["user_agent.original"], attributes["user_agent"])
          - delete_matching_keys(attributes, "^(client_address|user_name|time|method|path|query|protocol_version|status|size|referer|user_agent)$")
  batch:
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://your-backend:4318

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [filelog]
      processors: [memory_limiter, transform/semconv, batch]
      exporters: [otlphttp]

LinkMesh

LinkMesh has no built-in template for this job. Add it as a custom source in raw mode, with filelog as the component type and this body - the same lines as under filelog in the collector config:

Custom source body (filelog)
include: [/var/log/apache2/access.log]
start_at: end
storage: file_storage
operators:
  - type: regex_parser
    regex: '^(?P<client_address>\S+) \S+ (?P<user_name>\S+) \[(?P<time>[^\]]+)\] "(?P<method>[A-Z]+) (?P<path>[^ ?"]*)(?:\?(?P<query>[^ "]*))? HTTP/(?P<protocol_version>[0-9.]+)" (?P<status>\d{3}) (?P<size>\d+|-) "(?P<referer>[^"]*)" "(?P<user_agent>[^"]*)"'
    timestamp:
      parse_from: attributes.time
      layout: '%d/%b/%Y:%H:%M:%S %z'
    on_error: send

How custom components are validated and rolled out: create a custom source.

Download config.yamlValidated with otelcol-contrib v0.161.0 on 2026-10-26

Complete config files, each checked with the collector's own validate command (otelcol-contrib v0.161.0). Replace https://your-backend:4318 with your own OTLP endpoint before you run it.

Before and after

Example telemetry in, and what this exact config sent out when we ran it in otelcol-contrib v0.161.0. CI re-runs it on every change and fails if the output differs. Changed values are highlighted.

2 log records in, 2 log records out

  1. Before

    body
    203.0.113.7 - - [22/Oct/2026:10:15:02 +0200] "GET /index.html HTTP/1.1" 200 2326 "https://www.example.ch/" "Mozilla/5.0 (Macintosh) Safari/605.1.15"

    After

    body
    203.0.113.7 - - [22/Oct/2026:10:15:02 +0200] "GET /index.html HTTP/1.1" 200 2326 "https://www.example.ch/" "Mozilla/5.0 (Macintosh) Safari/605.1.15"
    timestamp
    2026-10-22T08:15:02.000Z
    attributes.log.file.name
    input.log
    attributes.client.address
    203.0.113.7
    attributes.http.request.method
    GET
    attributes.url.path
    /index.html
    attributes.network.protocol.name
    http
    attributes.network.protocol.version
    1.1
    attributes.http.response.status_code
    200
    attributes.http.response.body.size
    2326
    attributes.http.request.header.referer
    ["https://www.example.ch/"]
    attributes.user_agent.original
    Mozilla/5.0 (Macintosh) Safari/605.1.15
  2. Before

    body
    198.51.100.23 - - [22/Oct/2026:10:15:05 +0200] "GET /logo.png HTTP/1.1" 304 - "https://www.example.ch/index.html" "Mozilla/5.0 (Macintosh) Safari/605.1.15"

    After

    body
    198.51.100.23 - - [22/Oct/2026:10:15:05 +0200] "GET /logo.png HTTP/1.1" 304 - "https://www.example.ch/index.html" "Mozilla/5.0 (Macintosh) Safari/605.1.15"
    timestamp
    2026-10-22T08:15:05.000Z
    attributes.log.file.name
    input.log
    attributes.client.address
    198.51.100.23
    attributes.http.request.method
    GET
    attributes.url.path
    /logo.png
    attributes.network.protocol.name
    http
    attributes.network.protocol.version
    1.1
    attributes.http.response.status_code
    304
    attributes.http.request.header.referer
    ["https://www.example.ch/index.html"]
    attributes.user_agent.original
    Mozilla/5.0 (Macintosh) Safari/605.1.15

The gotcha

Apache's %b writes a dash, not 0, when no body was sent - every 304 and most HEAD requests. Int("-") fails: in our test, without the guard on that statement, the collector logged a "failed to execute statement" warning for the 304 line (and would for every one on a busy server), and the size attribute was simply missing. The where clause skips the dash instead.

Manage this across a fleet

LinkMesh manages collector configuration across a fleet from one self-hosted control plane, so a change like this is made once rather than host by host. The first 25 Collectors are free after a no-card registration in the OpenSight Customer Portal (5 without one).

Doing this in LinkMesh: Sources