node.kubernetes.io/unreachable: The node controller cannot reach the node and has tainted it

This taint is applied automatically when a node's Ready condition becomes Unknown. Pods tolerate it for a default of five minutes, after which they are evicted and rescheduled.

Applies to: All Kubernetes versions

What it means

The node controller applies node.kubernetes.io/unreachable with the NoExecute effect when it stops hearing from a node. Every pod gets an automatic toleration for it with tolerationSeconds: 300, so nothing moves for five minutes; after that, the taint evicts them. This delay is a deliberate trade-off. Too short, and a brief network blip causes a cluster-wide reshuffle. Too long, and workloads stay down on a node that is genuinely gone. Five minutes suits most clusters and suits latency-sensitive workloads poorly, which is why the toleration can be set explicitly per pod.

Most common causes

• The node lost network connectivity to the control plane.
• The kubelet crashed or was stopped.
• The machine was terminated abruptly, as with a spot instance reclaim.
• The node is so overloaded it cannot complete status updates.
• A control plane problem preventing heartbeats from being recorded.
• An expired kubelet certificate, so heartbeats are rejected.

How to diagnose it

1. Check the node's taints and conditions: kubectl describe node NODE.
2. Look at the last heartbeat timestamp to see when contact was lost.
3. Determine whether the machine still exists at the infrastructure layer.
4. Check whether the workloads on it are still serving traffic — they often are, which changes what is safe to do.
5. Check whether several nodes are affected at once, which points at the network or the control plane.

How to fix it

1. Restore connectivity or restart the kubelet.
2. Delete the node object if the machine is permanently gone, which releases its pods immediately rather than waiting for timers.
3. Set a shorter tolerationSeconds for the unreachable and not-ready taints on latency-sensitive workloads, accepting more churn during transient failures.
4. Ensure workloads run under a controller so replacements are scheduled automatically.
5. Use a PodDisruptionBudget and spread constraints so losing one node does not take a service below capacity.

Notes

Containers on an unreachable node usually keep running, so during those five minutes a workload can be serving from a node the control plane considers lost. For anything requiring at-most-one semantics, that window is real and must be handled by the application or by StatefulSet guarantees rather than assumed away.

Related

• Unknown — The control plane lost contact with the node
• NotReady — The node is not accepting work

Sources

• Kubernetes documentation — Taints and Tolerations: taint-based evictions
• Kubernetes documentation — Nodes: node controller
• Kubernetes documentation — Node Status