failed to stop container: The runtime could not terminate a container

A container that will not stop blocks pod deletion, node drains, and anything waiting on the pod's identity. It is usually a process wedged in an uninterruptible state rather than one ignoring signals.

Applies to: All Kubernetes versions

What it means

Stopping a container means sending SIGTERM, waiting for the grace period, then SIGKILL. SIGKILL cannot be ignored — but it also cannot interrupt a process blocked in an uninterruptible kernel wait, which is what happens when a process is stuck on I/O to a device or a network filesystem that is not responding. The container therefore stays in the runtime's state, the pod stays Terminating, and no amount of retrying helps until the underlying I/O completes or its device is cleared. This is the mechanism behind most pods that will not go away, and it explains why force-deleting the pod object does not actually stop anything.

Most common causes

• A process blocked in uninterruptible I/O, typically on an unresponsive network filesystem or storage device.
• A hung NFS mount inside the container.
• A kernel-level device error leaving a process stuck.
• The container runtime being unresponsive or overloaded.
• A very long termination grace period, which looks like a hang but is not.
• A process in a zombie state whose parent is not reaping it.

How to diagnose it

1. Check the process state on the node: a state of D in ps output confirms uninterruptible sleep.
2. Read the runtime log for the failed stop: journalctl -u containerd.
3. Check for hung mounts on the node, and test reachability of any network filesystem the container uses.
4. Check the pod's grace period before concluding anything: kubectl get pod POD -o jsonpath='{.spec.terminationGracePeriodSeconds}'.
5. Check the node's kernel log for device or filesystem errors.

How to fix it

1. Restore the unresponsive storage. A process in uninterruptible sleep resumes and exits once its I/O completes.
2. Clear the hung mount at the node level if the server cannot be restored.
3. Restart the container runtime if it is the component that is stuck rather than the process.
4. Reboot the node as a last resort — a process in uninterruptible sleep on a dead device sometimes leaves no other option.
5. Use mount options with finite timeouts on network filesystems so a dead server produces an error rather than an indefinite block.

Notes

Force-deleting the pod removes the API object while the container keeps running. For a StatefulSet member or anything else with at-most-one semantics, that is precisely the situation the identity guarantees exist to prevent, so it is worth confirming the process is actually gone rather than assuming.

Related

• Terminating — Pod stuck in deletion
• FailedUnmount — A volume could not be unmounted from the pod

Sources

• Kubernetes documentation — Pod Lifecycle: termination of pods
• Kubernetes documentation — Debugging Kubernetes nodes with crictl
• Kubernetes documentation — Force Delete StatefulSet Pods