System OOM encountered: The kernel's OOM killer acted at the node level, not on a single container's limit

This event means the node itself ran out of memory and the kernel killed a process to recover. Unlike a container OOM kill, the victim may be a pod that was well within its own limits.

Applies to: All Kubernetes versions, Linux nodes

What it means

There are two distinct kinds of out-of-memory kill in Kubernetes. A container OOM kill happens when a cgroup exceeds its own memory limit, and the victim is that container — reported as OOMKilled with exit code 137. A system OOM happens when the node as a whole exhausts memory, and the kernel picks a victim by its own scoring, which can be any process on the node including one belonging to a well-behaved pod. The kubelet records this as a SystemOOM event on the node. Reaching this point usually means eviction did not act in time, which happens when memory is consumed faster than the kubelet's monitoring interval can detect.

Most common causes

• Node memory exhausted by the sum of all workloads, with eviction unable to keep up.
• A workload allocating memory very rapidly, faster than the eviction loop's polling interval.
• Insufficient systemReserved and kubeReserved, so scheduling consumed memory the system needed.
• Containers with no memory limits, which can grow until the node fails.
• Kernel or system daemon memory usage that Kubernetes does not account for.
• Memory used outside pod cgroups, such as some kinds of page cache pressure.

How to diagnose it

1. Look for the event on the node: kubectl get events --field-selector reason=SystemOOM.
2. Read the kernel's own record on the node: dmesg -T | grep -i 'out of memory', which names the victim and its score.
3. Check which pods were affected and whether they were within their limits — a victim inside its limit confirms a system OOM rather than a container one.
4. Review the node's memory reservations against actual system usage.
5. Identify workloads with no memory limits, which are the usual precondition.

How to fix it

1. Set memory limits on every container so no workload can grow into the node's reserve.
2. Configure systemReserved and kubeReserved to reflect what the node's own processes actually use.
3. Set requests accurately so the scheduler does not overcommit the node in the first place.
4. Reduce pod density on nodes that hit this repeatedly.
5. Investigate any workload that allocates in large sudden bursts, since that is what outruns eviction.

Notes

The distinction between a container OOM kill and a system OOM matters when assigning blame. A container OOM kill is the workload exceeding what it asked for. A system OOM is a capacity planning failure, and the process the kernel chose to kill is often not the one responsible.

Related

• OOMKilled — Killed by the kernel for exceeding the memory limit
• MemoryPressure — The node is low on available memory

Sources

• Kubernetes documentation — Node-pressure Eviction
• Kubernetes documentation — Reserve Compute Resources for System Daemons
• Kubernetes documentation — Resource Management for Pods and Containers