Resume a log file after a restart

A File Tail source with Resume After Restart on remembers how far into each file it has read. A restart, an upgrade or a config push then carries on from where it stopped, instead of losing whatever was written while the collector was down.

Kubernetes Pod Logs sources use the same mechanism and behave identically — everything on this page applies to both.

The position is not kept by LinkMesh. It is kept on the collector host, by the collector, in a directory. Most of this page is about which directory.

The short version

Leave Checkpoint Directory empty. The collector then uses a directory it already owns on every shape we ship, and there is nothing to arrange.

Turn it on

1. Open the collector, go to the Inputs tab, and edit your File Tail or Kubernetes Pod Logs source.
2. Switch on Resume After Restart.
3. Leave Checkpoint Directory empty unless you have a reason not to — see Choosing your own directory.
4. Save. The collector picks up the change on its next config apply.

What it is not: start_at

These two settings are constantly confused, and they answer different questions:

• start_at decides what happens to a file the collector has never seen before.
• the checkpoint decides what happens to one it has.

	without the checkpoint	with the checkpoint
start_at: end	after every restart, whatever was written while the collector was down is lost	nothing is lost; a brand-new file still starts at its end
start_at: beginning	every restart re-reads every matched file from the top, duplicating everything	nothing is lost and nothing is duplicated; a brand-new file is still read in full

So start_at: end without a checkpoint is the combination that quietly loses data across every restart, and start_at: beginning without one is the combination that quietly duplicates it.

Where the position lives

One store per source, in a directory named after the source, so a directory on disk tells you which source owns it.

Linux host, otelcol-contrib (OpAMP)

Default location	/var/lib/otelcol-supervisor/filelog-storage/
Runs as	root
Anything to arrange?	No. systemd creates and owns the parent state directory
Survives a restart or reboot	Yes
Survives an upgrade	Yes — the installer replaces the binaries and leaves the state directory alone

Removing the service unit, or running systemctl clean, does delete it.

Linux host, Grafana Alloy

Default location	inside Alloy’s own data directory, which for the apt/rpm package is under /var/lib/alloy/data
Runs as	the unprivileged alloy user the package creates
Anything to arrange?	No. The path is inside the directory Alloy’s unit already owns — which is why no directory is named on this shape
Survives a restart or reboot	Yes
Survives an upgrade	Yes — apt/dnf keep package state. A purge deletes it

Kubernetes

On both Kubernetes shapes the position is held on the node, not inside the pod:

Shape	On the node
otelcol DaemonSet	/var/lib/linkmesh/otelcol/filelog-storage/
Alloy Helm chart	/var/lib/linkmesh/alloy

That is deliberate. The read position describes files on that node, so only a node-local volume brings it back to the pod that reads those same files.

It therefore survives a pod being replaced — a restart, a node drain, an image bump — as long as the replacement pod lands on the same node, which is what a DaemonSet does. Verified by deleting the pod mid-write: every line arrived, none twice.

Collectors installed before this shipped

A collector whose manifest predates node-local storage keeps its old in-pod volume and loses its position whenever the pod is replaced. Re-apply the manifest from the Add Collector wizard to fix it. Re-applying replaces the pod, so the position held on the old volume is not carried over — a one-off gap at the switch, and then it is durable.

A pod that lands on a different node reads that node’s files and uses that node’s positions, which is correct: they are different files.

Uninstalling leaves the node directory behind. Remove it by hand if the disk matters.

Docker

The position lives under whatever storage path the container is started with, inside the container. Mount that path as a volume, or it is lost every time the container is replaced.

Choosing your own directory

Setting Checkpoint Directory overrides the default. It must be an absolute path — a relative one is refused when you save, because the collector’s working directory is not something you can see from the UI.

An unwritable directory stops the whole collector

If the collector cannot create or write the directory, it refuses to start with that config, naming the path. Every other source and destination on that collector stops with it. An otelcol collector is then put back on the last config it applied and runs again within seconds, but without your change, until the directory is fixed — see When the collector cannot load it.

That is intentional. Starting anyway would give you a collector that looks healthy, ships data, and silently is not doing the one thing you turned on — and you would find out weeks later, as a gap in the logs during an incident.

LinkMesh cannot check the path before you save it: the server has no access to the collector’s filesystem, and a guess would trade a loud accurate failure for a quiet inaccurate green.

You will see something like this in the collector’s own log:

Error: failed to build extensions: failed to create extension
"file_storage/input_src_2f7cdf0d": mkdir /readonlyzone/state: read-only file system

Fix the permissions, or clear the field to fall back to the default.

Turning it off, moving it, deleting the source

• Turning it off stops the collector reading and writing the store. It does not delete it. Turning it back on resumes from the position last recorded — which may be a long way behind if time has passed.
• Changing the directory starts a new, empty store. The source is then a file the collector has never seen, so start_at decides what happens next. Move the old directory’s contents alongside the change if you want the position to follow it.
• Deleting the source leaves the directory on the host. Nothing reads it again; remove it by hand if the disk matters.

The other half: undelivered data

The checkpoint records how far the collector has read. It says nothing about what it has read but not yet delivered.

The receiver records its offset after handing records into the pipeline, not after a successful export — so anything still sitting in a destination’s queue when the collector stops is gone, and the checkpoint cannot cover it by construction. Turning on only the checkpoint is the common mistake.

If what you mean is “do not lose my data”, turn on Durable Delivery on the destination as well. It keeps the pending queue on disk, in its own directory next to this one, on all the same shapes. It is on by default for destinations created from now on, and off for ones that already existed.

Custom destinations

A custom destination (a Raw YAML body) supports Durable Delivery too, with the same four settings a built-in destination has: durable_queue, queue_storage_directory, queue_size and block_on_overflow. They go in the destination’s defaultConfig, or an activation’s configOverrides, beside the body — never in it. LinkMesh then adds the disk-backed storage and points the exporter’s sending_queue at it, exactly as for a built-in destination. In the Raw YAML editor they are on the Body step, under Durable Delivery, with the same labels as on a built-in destination. For an exporter type that cannot have it, the editor says why instead of offering the switch.

Two things differ from a built-in destination:

• It works for any exporter whose otelcol component has a sending_queue. An exporter without one, or a type LinkMesh has no settings list for, is refused with invalid_durable_queue.
• The body may keep its own sending_queue tuning, such as num_consumers. It may not also set storage, block_on_overflow, enabled: false, or a queue_size when the destination sets one — each of those would be set twice. That is refused with durable_queue_raw_body_conflict.

The other way to get the same result is to declare a file_storage extension on the collector yourself and write sending_queue: {storage: ${ext:<label>}} in the body. See Add a collector extension. Use one or the other on a destination, not both.

See also

• Source — where activations live and how they are stored
• Destination — where Durable Delivery is configured
• Upgrade collectors — what an upgrade does and does not touch