Probe: HTTP 503: The application answered the probe and reported itself unhealthy

An HTTP probe treats any status outside 200–399 as a failure. A 503 in particular means the application is running, received the request, and deliberately said it is not healthy — which is the most informative probe failure there is.

Applies to: All Kubernetes versions

What it means

An httpGet probe succeeds when the response status is greater than or equal to 200 and less than 400. Anything else fails. A 503 is therefore not a network problem at all: the application is up, it handled the request, and its own health logic decided the answer was no. That makes it the one probe failure where the application is the authority and the fix belongs inside it. The corollary is that you have to know what the endpoint actually checks. Many frameworks ship a health endpoint that aggregates every registered dependency, so a single degraded cache turns into a 503, which — if wired to a liveness probe — restarts a process that was working.

Most common causes

• The application's own health logic reports a dependency as down.
• The application is shutting down and correctly reports 503 to drain traffic.
• A framework health endpoint aggregating dependency checks, where one non-critical dependency fails the whole response.
• A dependency that is slow rather than down, tripping a timeout inside the health check.
• The probe hits a path served by a proxy or ingress that returns 503 when no backend is available, rather than the application itself.
• A 401 or 403 rather than 503, when the health endpoint requires authentication the probe does not supply.

How to diagnose it

1. Call the endpoint directly and read the body: kubectl exec POD -- curl -i localhost:PORT/healthz. Most frameworks report which check failed.
2. Check the application logs at the same timestamp for the failing dependency.
3. Determine what the endpoint checks. If it is a framework default, it may be checking more than you intended.
4. Confirm the probe reaches the application and not a proxy in front of it.
5. For 401 or 403, check whether the endpoint requires credentials — probes send no authentication by default, though httpHeaders can add them.

How to fix it

1. Separate the endpoints. A liveness endpoint should check only that the process is functioning; a readiness endpoint may check dependencies. Pointing both at the same aggregate check is the root cause of most damage here.
2. Exclude non-critical dependencies from the health aggregation, or downgrade them so they degrade rather than fail.
3. Exempt the health path from authentication, or supply the credential through httpHeaders.
4. If the application returns 503 during shutdown, that is correct — ensure it is the readiness probe reading it, so the pod is drained rather than restarted.

Notes

Redirects in the 300 range count as success, because the range is 200–399 inclusive of 3xx. An endpoint that redirects to a login page therefore passes the probe while telling you nothing about health.

Related

• Readiness probe failed — The pod was removed from service endpoints
• Unhealthy — The event emitted on any probe failure

Sources

• Kubernetes documentation — Configure Liveness, Readiness and Startup Probes
• Kubernetes documentation — Pod Lifecycle: container probes