exec probe failed: The probe command ran inside the container and exited non-zero

An exec probe succeeds only when its command exits 0. A failure means the command ran and returned an error, or could not be run at all — and exec probes are the most expensive probe type to run frequently.

Applies to: All Kubernetes versions

What it means

An exec probe runs a command inside the container and treats exit status 0 as healthy. This makes it flexible — it can check anything a command can check — but it has costs the other probe types do not. Every execution spawns a process inside the container, which consumes CPU and memory that count against the container's limits, and on a container running near its limit a frequent exec probe can itself contribute to the pressure it is meant to detect. The failure modes are also broader: as well as the command reporting a genuine problem, the command may not exist in the image, may not be executable, or may exceed the probe timeout and be counted as a failure.

Most common causes

• The command genuinely detected an unhealthy condition and exited non-zero.
• The command does not exist in the image — a probe using curl or wget in a minimal or distroless image is the classic case.
• The command exists but is not executable by the container's user.
• The command takes longer than timeoutSeconds.
• A shell form was used in an image with no shell.
• The command writes to stdout and blocks, or waits for input that never comes.

How to diagnose it

1. Run the command manually and check its exit status: kubectl exec POD -- sh -c 'COMMAND; echo exit=$?'.
2. If the image has no shell, run the command in exec form directly to see whether it exists at all.
3. Read the event message — a missing binary produces an OCI exec error rather than an ordinary non-zero status, and the wording differs.
4. Time the command. If it is near timeoutSeconds, intermittent failures under load are guaranteed.
5. Check the container's memory headroom, since each probe execution needs some.

How to fix it

1. Prefer httpGet or tcpSocket probes where they can express the same check. They cost nothing inside the container and have narrower failure modes.
2. Ensure the probe command and any interpreter it needs are present in the image, and remember that distroless images have neither a shell nor common utilities.
3. Keep the command fast and give it a realistic timeout.
4. Lengthen periodSeconds for exec probes so the overhead is proportionate.
5. Make the command's failure mode explicit — it should exit non-zero only for the condition you mean, not for any incidental error.

Notes

Kubernetes has a native grpc probe type as well, which is the right choice for gRPC services and avoids the common workaround of shipping grpc_health_probe into the image purely so an exec probe can call it.

Related

• Unhealthy — The event emitted on any probe failure
• Exit code 127 — Command not found

Sources

• Kubernetes documentation — Configure Liveness, Readiness and Startup Probes
• Kubernetes documentation — Pod Lifecycle: container probes