cannot allocate memory: An allocation failed inside the container before any OOM kill happened

ENOMEM means the kernel refused an allocation. In a container this usually means the cgroup limit was reached on an allocation the kernel could fail rather than kill for — so the process survives to report it.

Applies to: All Kubernetes versions, Linux nodes

What it means

Not every memory exhaustion ends in an OOM kill. Some allocations can be failed cleanly, and the kernel returns ENOMEM to the process instead of invoking the OOM killer. The application then either handles it or crashes on its own terms, which is why this error appears in application logs while the container's status shows no OOMKilled reason. It also appears for allocations that are not about the memory limit at all — creating a thread when PIDs are exhausted, or mapping memory when a virtual address space limit is reached. The distinguishing feature from an OOM kill is simply that the process was alive to write the message.

Most common causes

• The container's memory cgroup limit was reached on a failable allocation.
• Thread creation failing because of PID limits, which some runtimes report as a memory error.
• The node itself being out of memory and overcommit settings refusing the allocation.
• A single allocation larger than the container's entire limit.
• Address space limits reached in a process mapping many regions.
• Memory fragmentation preventing a large contiguous allocation even with free memory available.

How to diagnose it

1. Check whether the container was ever OOM-killed: kubectl describe pod POD. A clean status with ENOMEM in the logs is the distinguishing case.
2. Compare usage against the limit over time rather than at a point: a metrics dashboard shows whether it is at the ceiling.
3. Check PID limits if the failure is around thread creation.
4. Check node-level memory and overcommit settings if several pods are affected.
5. Look at the size of the failing allocation, if the application reports it.

How to fix it

1. Raise the container's memory limit if usage is genuinely at the ceiling.
2. Reduce the application's memory footprint, or the size of individual allocations.
3. Raise PID limits if thread creation is the failing operation.
4. Configure the runtime's heap relative to the cgroup limit rather than to the node's memory.
5. Add node capacity if the node itself is exhausted.

Notes

The absence of an OOMKilled reason does not rule out a memory problem. An application that handles allocation failure gracefully will report ENOMEM and keep running in a degraded state, which is easy to misread as an unrelated application error.

Related

• OOMKilled — Killed by the kernel for exceeding the memory limit
• System OOM encountered — The node itself ran out of memory

Sources

• Kubernetes documentation — Resource Management for Pods and Containers
• Kubernetes documentation — Assign Memory Resources to Containers and Pods
• Kubernetes documentation — Node-pressure Eviction