Fix Docker cgroupv2 Errors in CI Runners

DEV Community

Docker cannot enter cgroupv2, and cgroup mount errors in CI

Docker cannot enter cgroupv2, and the cgroup mount errors beside it, mean the runtime could not place your container into a control group, which on a CI runner nearly always means a daemon running nested inside another container without the v2 controllers delegated to it. Your image and your command are not involved: the container has nowhere to be accounted. A nested daemon can only place containers in cgroups its own environment was given. Delegation is what decides that, and it is not automatic. What this error means A container refuses to start and the message points at /sys/fs/cgroup rather than at anything you wrote, often with the phrase "with domain controllers" and a note that the state is invalid. It appears in Docker-in-Docker jobs, in Kubernetes-in-Docker tools, and in build engines that run their own daemon in a container, and it does not appear on a plain hosted runner doing a plain docker run. We did not reproduce it: it needs a nested daemon in a particular broken delegation state, and building one to photograph it would teach you less than the delegation rules below. The line below is quoted in full from containerd/containerd#6659, ctr prefix and line numbers included, because the clause that names the cause is in the middle of it: applying cgroup configuration for process. The Go file and line numbers move between runc releases, so match on that clause rather than on them. ```The line containerd/containerd#6659 reports (illustrative, not a recorded run) ctr: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/default" with domain controllers -- it is in an invalid state: unknown

Common causes

Controllers were never delegated to the nested environment

The dominant cause. A daemon inside a container gets a cgroup subtree, and the controllers it needs have to be enabled in that subtree by whatever created it. When they are not, the runtime cannot create the child cgroup for a new container, and reports that it cannot enter the cgroup it was given.

The cgroup mount is missing or read-only inside the container

A nested daemon needs /sys/fs/cgroup present and writable for its own subtree. A hardened container, a read-only mount, or a sandbox that hides the hierarchy produces mount errors in the same family, with the same effect: nothing can be placed in a cgroup.

The cgroup driver does not match the host

On a v2 host the default driver is systemd. A nested runtime or a tool configured for cgroupfs will fight the systemd hierarchy, and in our experience this is the cause when the error changes shape between runs rather than being stable.

The runtime predates proper cgroup v2 support

Older runc and containerd releases handle the unified hierarchy badly, which is what containerd/containerd#6659 is about. A pinned old runtime in a custom image, or a base image nobody has refreshed, is enough to reproduce a bug that current versions do not have.

How to fix it

Use the daemon the runner already has

1. Drop the docker:dind service and run Docker commands directly in the job.
2. If a tool insists on its own engine, check whether it has a mode that reuses the host daemon.
3. Keep nesting for cases that genuinely need isolation between builds.

```.github/workflows/ci.yml
- run: docker info --format '{{.CgroupVersion}} {{.CgroupDriver}}'
- run: docker build -t app .
Give the nested daemon the cgroup environment it needs When you do nest, run the inner daemon with the host cgroup namespace and the privileges to manage its subtree, and mount the hierarchy so it is writable. This is a deliberate reduction in isolation, so scope it to the job that needs it rather than making it the default. docker run --privileged --cgroupns=host \
- v /sys/fs/cgroup:/sys/fs/cgroup:rw \
docker:29-dind
Align the cgroup drive...

Powered by Readybot.io