FailedDetachVolume: The volume could not be detached from a node

FailedDetachVolume means the control plane asked the storage system to release a disk from a node and it did not. Until it succeeds, no pod on another node can use that volume.

Applies to: All Kubernetes versions using attachable volume types

What it means

Detach is the mirror of attach, performed by the attach-detach controller after the last pod using a volume on a node has gone. When it fails, the volume stays bound to a node that no longer needs it, and any pod scheduled elsewhere blocks with a multi-attach error. The two usual reasons are that the volume is still in use — a process on the node still has the filesystem open, so the unmount could not complete — or that the node itself is unreachable, in which case nothing on the node can confirm the release and the controller will not force it without risking data corruption.

Most common causes

• A process on the node still has the filesystem open, so the unmount fails and the detach cannot proceed.
• The node is unreachable, so the controller has no confirmation the volume is safe to release.
• A container that was force-deleted at the API level but is still running on the node.
• The CSI driver's node plugin is not running on that node, so unmount cannot be performed.
• The storage backend is rejecting the detach, for example because of an in-progress snapshot or backup operation.
• The device is in an error state at the kernel level, common with network block devices after a connectivity loss.

How to diagnose it

1. Read the controller's view: kubectl get volumeattachment shows the attachment still present.
2. Check the attach-detach controller's messages in the kube-controller-manager log.
3. On the node, find what is holding the mount: lsof +D /var/lib/kubelet/pods/… or fuser -vm MOUNTPOINT.
4. Check whether the CSI node plugin is running there: kubectl get pods -n kube-system -o wide | grep NODE.
5. Check the node's own status and reachability.

How to fix it

1. Stop the process still holding the filesystem, or restart the container that owns it.
2. Restore the CSI node plugin on the node if it is missing or crashed.
3. If the node is permanently gone, delete the node object. The controller then treats the attachments as releasable.
4. Wait out any backend operation such as a snapshot that is blocking the detach.
5. For network block devices in an error state, clearing the device at the node level may be required before the detach can complete.

Notes

Force-detaching at the storage backend while a node might still be writing is how filesystems get corrupted. The conservative behaviour here is deliberate, and working around it should be a considered decision rather than a reflex.

Related

• Multi-Attach error — The volume is already attached to another node
• FailedUnmount — A volume could not be unmounted from the pod

Sources

• Kubernetes documentation — Persistent Volumes
• Container Storage Interface specification
• Kubernetes documentation — Nodes