Fixing Fork EAGAIN Failures in CI Builds

DEV Community

Fork retry resource temporarily unavailable CI failures

Fork retry resource temporarily unavailable CI failures mean the kernel refused to create a new process with EAGAIN, and EAGAIN in fork(2) is a limit on the number of processes and threads, not a shortage of memory. Count the threads your build is running and compare that against ulimit -u and the cgroup PID limit before you touch the runner size. The two failure paths in bash make_child. The retry loop runs only while errno is EAGAIN, which is why an out-of-memory fork prints once and never prints "retry". What this error means The step prints the same line several times and then a shorter version of it, after which almost every command fails because nothing can start a child process. The repetition is not noise: bash retries on this specific error and prints once before each retry, with a doubling sleep in between, so a step can sit there for several seconds looking hung before it gives up. The last line drops the retry: because it comes from a different branch of the same function, the one that runs after the loop has stopped trying. The prefix is worth reading too. In a workflow run: step it is not the word bash but the path of the temporary script the runner wrote, because bash names the error after BASH_SOURCE[0] whenever it is not interactive. ```Reconstructed from sys_error in bash make_child (jobs.c) fork: retry: Resource temporarily unavailable fork: retry: Resource temporarily unavailable fork: retry: Resource temporarily unavailable fork: Resource temporarily unavailable

Common causes

Nested parallelism multiplied out

The common case, and it is arithmetic rather than a bug. A test runner given one worker per core starts a build tool that is also given one job per core, and each of those starts a language runtime with its own thread pool. On a 4 vCPU runner that is not twelve threads, it is four times four times some pool size, and it can reach four figures without anything looking wrong in the workflow file. The ceiling is reached by multiplication, which is why the same command runs fine on a laptop with a much higher limit.

A container or job with a low pids limit

Running your build inside a container adds a second ceiling. The cgroup PIDs controller enforces pids.max independently of ulimit -u, and it is frequently the lower of the two, so a build that has plenty of headroom by the shell limit is stopped by the cgroup anyway. This is the version that appears when a job is moved into a container and nothing else changes, and it will not reproduce outside that container.

Something is leaking processes or threads

A step that starts a background service on every iteration of a loop, a test suite that spawns a helper per test and never waits for it, or a tool that leaves zombies behind will climb steadily toward the limit rather than jumping to it. The signature is that the failure happens late in a long step rather than immediately, and that it moves earlier as the suite grows. Counting with ps -eLf | wc -l at the start and end of the step turns this from a theory into a number.

The limit is genuinely too low for the work

Some builds legitimately need thousands of threads, and a runner image or container configured conservatively will refuse them. This is the least common of the four and the only one where raising the limit is the right answer rather than a way of postponing the question, so establish it by measurement before you conclude it.

How to fix it

Measure the ceiling and the count, in that order

1. Print ulimit -u and cat /sys/fs/cgroup/pids.max in the failing step. The effective ceiling is the lower of the two, and inside a container it is usually the second.
2. Count threads rather than processes with ps -eLf | wc -l, because RLIMIT_NPROC counts threads and ps -ef does not show them.
3. Sample the count at the start of the step and again just before the failure. A number that climbs is a leak; a number that jumps is parallelism.

```.github/wor...

Powered by Readybot.io