KubeErrors

unknown service runtime.v1alpha2.RuntimeService: The kubelet and the container runtime disagree about which CRI version to speak

This means the kubelet is calling a CRI API version the runtime does not serve. It is a version compatibility problem between the kubelet and containerd or CRI-O, not a configuration error.

Applies to: Kubernetes 1.26 and later with older runtimes, or the reverse

What it means

The Container Runtime Interface has versioned service definitions, and the v1alpha2 API was removed in favour of v1. A kubelet that only speaks v1 talking to a runtime that only serves v1alpha2, or the reverse, produces unknown service and the node cannot run pods at all. This is almost always the result of upgrading one side without the other — a Kubernetes upgrade on a node whose containerd is too old is the usual sequence. The node reports NotReady and every operation involving containers fails, so the impact is total rather than partial.

Most common causes

How to diagnose it

  1. Check both versions on the node: kubelet --version and containerd --version, or crio --version.
  2. Read the kubelet log, where the error appears directly: journalctl -u kubelet | grep -i 'unknown service'.
  3. Check the node's reported versions: kubectl get node NODE -o jsonpath='{.status.nodeInfo}'.
  4. Determine whether only some nodes are affected, which confirms a partial upgrade.
  5. Check the runtime's supported CRI versions against the Kubernetes version's requirements.

How to fix it

  1. Upgrade the container runtime to a version that serves the CRI API the kubelet requires. This is the correct fix.
  2. Downgrade the kubelet as a temporary measure if the runtime cannot be upgraded immediately.
  3. Rebuild node images so the kubelet and runtime versions are upgraded together.
  4. Include the container runtime in the upgrade plan for every Kubernetes upgrade — it is a dependency, not an independent component.
  5. Replace affected nodes with correctly built ones rather than patching them in place, where the node pool supports it.

Notes

This failure is total for the node: no pod can start, and existing pods cannot be managed. Because it appears immediately after an upgrade, it is worth verifying runtime compatibility before rolling a Kubernetes upgrade across a fleet rather than discovering it node by node.

Related

Sources

Pages on this site are written with AI assistance from the primary sources listed on each page, then checked against those sources before publishing.