Probe: timeout: The probe connected or tried to, and got no answer in time

A probe timeout — reported as context deadline exceeded or Client.Timeout exceeded — means the pod did not answer within timeoutSeconds. The default of one second is far too short for many real endpoints.

Applies to: All Kubernetes versions

What it means

When a probe exceeds timeoutSeconds, the kubelet records a timeout error. Unlike a refusal, this does not prove the packet reached a listener: a blocked route, a network policy dropping traffic silently, and an overloaded application all present identically. What makes timeouts particularly worth scrutinising is the default: timeoutSeconds defaults to 1. A health endpoint that queries a database, checks a cache, or serialises a status document can easily exceed one second under load — precisely when you least want the pod restarted. A probe that fails only under load, and restarts pods only when the system is busiest, is a reliability hazard dressed as a safety feature.

Most common causes

• timeoutSeconds left at the default of 1 for an endpoint that does real work.
• The health endpoint queries downstream dependencies and inherits their latency.
• The container is CPU-throttled at its limit and cannot schedule the request handler promptly.
• The application's event loop or thread pool is saturated, so the request queues behind real traffic.
• A network policy dropping the kubelet's probe traffic — dropped packets time out rather than being refused.
• Garbage collection pauses in a managed runtime exceeding the probe timeout.

How to diagnose it

1. Read the exact error: context deadline exceeded and Client.Timeout exceeded while awaiting headers both indicate a timeout rather than a refusal.
2. Measure the endpoint's real latency under load from another pod, not from an idle cluster.
3. Check CPU throttling on the container — throttled containers fail timing-sensitive probes first.
4. Check whether failures correlate with traffic peaks or with garbage collection pauses.
5. If a network policy is in place, confirm it permits traffic from the node to the pod.

How to fix it

1. Raise timeoutSeconds to something realistic, and raise failureThreshold so one slow response does not act.
2. Make the health endpoint cheap. It should not query dependencies or do meaningful work; a liveness endpoint in particular should answer from memory.
3. Raise the CPU limit, or remove it, if throttling is causing the latency.
4. Serve health endpoints on a separate port or thread pool so they are not queued behind application traffic.
5. Allow probe traffic explicitly in the network policy.

Notes

timeoutSeconds must be less than periodSeconds for the configuration to behave sensibly — otherwise probes overlap and the effective behaviour becomes hard to reason about.

Related

• Probe: connection refused — Nothing is listening on the probed port
• Liveness probe failed — The container was restarted as unhealthy

Sources

• Kubernetes documentation — Configure Liveness, Readiness and Startup Probes
• Kubernetes documentation — Network Policies
• Kubernetes documentation — Resource Management for Pods and Containers