too many open files: A process or the node hit its file descriptor limit

EMFILE or ENFILE means descriptors ran out — per process or system-wide. In Kubernetes the usual culprits are leaked connections, unbounded concurrency, or the inotify watch limit, which is a separate and easily exhausted resource.

Applies to: All Kubernetes versions, Linux nodes

What it means

Every socket, file, and pipe consumes a file descriptor, bounded per process by RLIMIT_NOFILE and system-wide by a kernel maximum. Containers inherit the limit from the runtime rather than from the image, so a container can hit a ceiling nobody set deliberately. The variant worth calling out separately is inotify: watching files consumes inotify instances and watches, both capped by node-level kernel parameters, and these are shared across every container on the node. A single application watching a large directory tree can exhaust them for everything else, producing this error in workloads that are doing nothing unusual.

Most common causes

• A connection or file descriptor leak in the application.
• Unbounded concurrency — one descriptor per request with no ceiling.
• A descriptor limit inherited from the runtime that is lower than the workload needs.
• inotify watch or instance limits exhausted at the node level, affecting all containers.
• Many containers on one node collectively exhausting the system-wide maximum.
• A file-watching tool, log shipper, or hot-reloading framework consuming watches.

How to diagnose it

1. Check the process's limit and usage: kubectl exec POD -- sh -c 'cat /proc/1/limits | grep files; ls /proc/1/fd | wc -l'.
2. Check node-wide usage: cat /proc/sys/fs/file-nr on the node.
3. Check inotify limits, which are separate: sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances.
4. Determine whether the count grows steadily, which indicates a leak, or spikes with load.
5. Check whether several unrelated pods on the same node are affected, which points at a node-level limit.

How to fix it

1. Fix the leak. Close connections and files, and set bounds on connection pools.
2. Raise the container runtime's descriptor limit if the workload legitimately needs more, since containers inherit it.
3. Raise the inotify limits on the node for workloads that watch many files. The defaults are low relative to what modern tooling uses.
4. Bound concurrency in the application so descriptor use is proportional to a limit you chose.
5. Reduce the number of file-watching workloads on a single node if inotify is the constraint.

Notes

inotify limits are node-wide and shared, so one pod exhausting them causes failures in unrelated pods that look like application bugs. It is worth checking the inotify counters before investigating any application that suddenly cannot watch files.

Related

• PIDPressure — The node is running out of process IDs
• connection reset by peer — An established connection was terminated abruptly

Sources

• Kubernetes documentation — Node-pressure Eviction
• Kubernetes documentation — Reserve Compute Resources for System Daemons
• Kubernetes documentation — Debug Running Pods