failed to pull the sandbox image: The runtime cannot fetch the pause image every pod needs

Every pod's sandbox uses a small pause image. If the node cannot pull it, no pod can start there at all — which makes this look like a total node failure rather than an image problem.

Applies to: All Kubernetes versions

What it means

Each pod sandbox runs a minimal container, conventionally called pause, whose only job is to hold the pod's namespaces open. It is configured in the runtime rather than in any pod spec, which is why nothing in your manifests mentions it. When the node cannot pull it, every pod fails at sandbox creation, and the events say the sandbox could not be created rather than naming an image you recognise. This is most often seen in air-gapped or restricted-registry environments, where every application image was mirrored and the pause image was overlooked because it appears in no manifest.

Most common causes

• An air-gapped cluster where the pause image was not mirrored into the internal registry.
• A registry policy or admission control blocking the default upstream registry.
• The runtime's configured sandbox image pointing at a registry the node cannot reach.
• Registry authentication required for the pause image, which the runtime is not configured to supply.
• Rate limiting on the public registry hosting the default image.
• A runtime upgrade that changed the default pause image to a version not present in the mirror.

How to diagnose it

1. Read the sandbox creation error: kubectl describe pod POD — the failure names the image being pulled.
2. Check what the runtime is configured to use, in containerd's configuration under the CRI plugin's sandbox image setting.
3. Check whether the image is present locally: crictl images | grep pause.
4. Try the pull by hand on the node: crictl pull IMAGE.
5. Check whether every pod on the node is failing, which distinguishes this from a single application image problem.

How to fix it

1. Mirror the pause image into the registry the cluster uses, and configure the runtime to pull it from there.
2. Configure registry credentials at the runtime level, since this pull happens outside any pod's imagePullSecrets.
3. Pre-load the image onto node images so it is present before the node joins.
4. Check the sandbox image setting after any runtime upgrade, since the default can change.
5. Allow the registry hosting the image through any egress policy.

Notes

Because the pause image is configured in the runtime and not in any manifest, it is invisible to every audit that reads Kubernetes objects. In air-gapped environments it is one of the most commonly missed dependencies, and its absence disables the node completely rather than partially.

Related

• FailedCreatePodSandBox — The runtime could not create the pod's network sandbox
• ImagePullBackOff — Kubernetes could not pull the container image

Sources

• containerd CRI plugin configuration
• Kubernetes documentation — Container Runtime Interface
• Kubernetes documentation — Images