Fix Docker Read-Only File System Errors in CI

DEV Community

Docker mkdir /var/lib/docker: read-only file system in CI

Docker mkdir /var/lib/docker: read-only file system means the daemon tried to create a directory under its own data root and the kernel refused because the filesystem holding it is mounted read-only. This is a host condition rather than anything about your image or your workflow, and the daemon is reporting it accurately from one level below where a job can reach. One read-only mount, three different operation words. The path is the constant and the verb is whatever the daemon tried. What this error means Pulls, builds and runs all fail together, with paths under /var/lib/docker in the message and an operation word at the front that changes depending on what the daemon was doing. One public report of a single broken host carries mkdir, unlinkat and open forms of the same failure within a few lines of each other, which is the tell: when the data root goes read-only, everything that writes there fails, and each failure names whichever call happened to hit it. ```The line abiosoft/colima#883 reports (quoted, not a recorded run) API error (500): mkdir /var/lib/docker/overlay2/a784aef8fbf3af0f2a38904f816e31b88e268056b829a6b35028d77864d4a4c2-init: read-only file system

Common causes

The kernel remounted the filesystem read-only after I/O errors

The most common cause on a machine that was working earlier. A failing virtual disk produces read or write errors, and the kernel protects the filesystem by refusing further writes. Every Docker operation that touches the data root fails from that moment, which is why the errors arrive in a burst rather than one at a time.

The host root filesystem is read-only by design

Immutable and hardened host images ship a read-only root, and a daemon whose data root still points at the default path under it has nowhere to write. In our experience this is the cause when the failure is present from the very first container on a freshly provisioned host rather than appearing partway through its life.

The data root was pointed at a mount that is not writable

A data-root set in daemon.json to a network share, a snapshot or a volume mounted with the ro option produces this immediately. The setting looks right in the config and the mount options are where the problem actually is, so checking the config alone will not find it.

A container was given the data root as a read-only bind

Tooling that mounts /var/lib/docker into a container to inspect it sometimes mounts it read-only, and a nested daemon handed that mount reports exactly this. Here the host is fine and the container is the one that cannot write, which a mount listing inside the container settles.

How to fix it

Find out whether the mount or the config is the problem

1. Ask where the daemon thinks its root is, rather than assuming the default path.
2. Read the mount options for that path with findmnt, which prints ro or rw plainly.
3. Check the kernel log, because a mount that went read-only will say why it did.

```Terminal
docker info --format '{{.DockerRootDir}}'
findmnt -no OPTIONS -T "$(docker info --format '{{.DockerRootDir}}')"
dmesg | grep -i -E 'remount|read-only|I/O error' | tail
Move the data root onto a volume you control On a self-hosted runner, give the daemon a dedicated writable volume rather than leaving it on whatever the host image provided. This survives host images that harden the root filesystem, and it gives you something you can size and monitor separately from everything else on the machine. { "data-root": "/mnt/docker-data" }
Remount read-write only when the kernel has not objected If the mount is read-only because of how it was mounted rather than because the kernel demoted it, remounting is the fix. If the kernel demoted it after I/O errors, remounting without repairing the filesystem first is how a bad disk becomes a corrupt one, so read the kernel log before you touch it. mount | grep /var/lib/docker
sudo mount -o remount,rw /var/lib/docker
Fail the job on an unwrit...

Powered by Readybot.io