Linux Too Many Open Files: Limits Explained
Resumo
Pixel shares a practical guide to Linux file descriptor limits and why services can hit the “Too many open files” error without a leak. It explains the difference between per-process EMFILE and system-wide ENFILE, where to inspect limits, and how to identify the right fix.
DEV Community
Too Many Open Files Is Not a Leak: Practical File Descriptor Limits on Linux
A service runs for three weeks and then starts returning 500s. The log has one line worth reading: OSError: [Errno 24] Too many open files
The reflex is to hunt for a leak. Sometimes that is the right answer. Often it is not. The process is simply holding as many descriptors as it is allowed to hold, and nobody ever checked what that number was. The error is precise about which limit was hit, but vague about why. Linux has at least four different ceilings that produce the same family of symptoms: the per-process soft limit, the per-process hard limit, the kernel's ceiling on that hard limit, and the system-wide file limit. Add select() and inotify and you have two more traps that produce equally confusing failures. Here is how the numbers actually fit together, how to reproduce the failure on purpose, and how to raise the right one without papering over a real leak. Two errors, two scopes The first useful split is the errno: errno Name Scope Counter to inspect 24 EMFILE Per process /proc/PID/limits 23 ENFILE System-wide /proc/sys/fs/file-nr EMFILE means this process tried to allocate a descriptor number at or above its RLIMIT_NOFILE. ENFILE means the kernel ran out of file handles globally, regardless of what any single process is allowed. Most "too many open files" incidents are EMFILE. ENFILE is rarer and usually points at a system-wide flood or a very low fs.file-max, not at one badly behaved service. A descriptor is a handle, not a file It is tempting to read "open files" as "files on disk". That is wrong, and it makes the counts confusing. A file descriptor is an index into a per-process table. Each entry points at an open file description in the kernel. The same description can be referenced by several descriptors after dup(), and a descriptor can refer to almost anything: regular files and directories TCP and Unix sockets pipes and FIFOs character and block devices epoll instances, eventfd, signalfd, timerfd inotify instances, pidfd, memfd files passed in from another process over a Unix socket ls -l /proc/$PID/fd
The listing is the actual table. A healthy web server might show a few log files, a listening socket, hundreds of socket:[...] entries, and a couple of anon_inode:[eventpoll] entries. All of them compete for the same numeric limit. Two consequences follow. First, fork() copies the descriptor table but shares the underlying open file descriptions. Closing a descriptor in the child does not close the parent's copy. A forking service can therefore have many processes each holding a descriptor for the same socket. Second, since Linux 4.5, RLIMIT_NOFILE also caps how many descriptors an unprivileged process may have "in flight" to other processes via SCM_RIGHTS on Unix sockets. That is an edge case, but it explains why a descriptor-passing daemon can hit EMFILE without a table that looks full. The soft limit is the one that bites Every process has two values for RLIMIT_NOFILE: soft limit — the value enforced right now hard limit — the ceiling the soft limit can be raised to ulimit -Sn # soft
ulimit -Hn # hard
An unprivileged process can lower either limit and can raise the soft limit up to the hard limit. Raising the hard limit requires CAP_SYS_RESOURCE. The kernel also refuses to set the hard limit above /proc/sys/fs/nr_open, which defaults to 1048576 and is enforced for privileged processes too (setrlimit(2), proc_sys_fs(5)). This is where systemd's default explains a lot of surprise: DefaultLimitNOFILE=1024:524288
That is the documented manager default in systemd-system.conf(5): a soft limit of 1024 and a hard limit of 524288 for processes started by units. A service that never overrides it starts with 1024 usable descriptors, even though ulimit -Hn in a login shell might show half a million. The application is allowed to raise its own soft limit to 524288. Some applications do that at startup; many do not, and they are the ones that fall over at roughly a thousand connections. One more detail ...
Powered by Readybot.io