volume node affinity conflict: The volume exists somewhere the pod cannot be scheduled

This scheduling failure means a bound volume has topology constraints — usually an availability zone — that no eligible node satisfies. The pod and its data are in different places and neither can move.

Applies to: All Kubernetes versions with topology-aware storage

What it means

A PersistentVolume can carry node affinity describing where it can be accessed from. Zonal block storage is bound to a single zone; local volumes are bound to a single node. When the scheduler evaluates a pod using such a volume, nodes outside that topology are rejected, and the FailedScheduling message includes node(s) had volume node affinity conflict. This is a hard constraint: the volume genuinely cannot be reached from those nodes. The usual origin is a volume provisioned before the scheduler knew where the pod would run — with volumeBindingMode: Immediate, the volume is created as soon as the claim exists, in whatever zone the provisioner picks, and the pod must then follow it.

Most common causes

• A StorageClass with volumeBindingMode: Immediate provisioning a volume in a zone with no capacity for the pod.
• A node pool scaled to zero, or drained, in the zone where the volume lives.
• A local volume whose node has been removed or made unschedulable.
• Node affinity or a node selector on the pod that conflicts with the volume's zone.
• A cluster autoscaler adding nodes only in zones that do not match the volume.
• A statically provisioned PersistentVolume whose node affinity was written incorrectly.

How to diagnose it

1. Read the scheduler's breakdown: kubectl describe pod POD and count how many nodes were rejected for this reason.
2. Find the volume's topology: kubectl get pv PV_NAME -o jsonpath='{.spec.nodeAffinity}'.
3. List node zones: kubectl get nodes -L topology.kubernetes.io/zone.
4. Check whether any schedulable node exists in the volume's zone at all.
5. Check the StorageClass's binding mode: kubectl get storageclass NAME -o jsonpath='{.volumeBindingMode}'.

How to fix it

1. Restore capacity in the volume's zone — scale a node pool there, or uncordon the node for a local volume.
2. Set volumeBindingMode: WaitForFirstConsumer on the StorageClass so future volumes are created where the pod is scheduled. This does not move existing volumes.
3. For an existing conflict, either move the data to a volume in a reachable zone or make the target zone schedulable. There is no in-place fix.
4. Remove pod affinity rules that conflict with the volume's location.
5. Configure the autoscaler to provision nodes in every zone your volumes can live in.

Notes

This is one of the few Kubernetes failures with no recovery path in the pod spec. The data has a physical location; either the compute goes to it, or the data is copied. Setting WaitForFirstConsumer from the start avoids the whole class of problem.

Related

• FailedScheduling — The scheduler could not find a suitable node
• PersistentVolumeClaim Pending — The claim has not bound to a volume

Sources

• Kubernetes documentation — Persistent Volumes: node affinity
• Kubernetes documentation — Storage Classes: volume binding mode
• Kubernetes documentation — Kubernetes Scheduler