logs are truncated or missing: Older log lines have been rotated away by the node

The kubelet rotates container logs at a size limit and keeps a bounded number of files. Output older than that is deleted from the node, so kubectl logs cannot return it no matter what flags are used.

Applies to: All Kubernetes versions

What it means

Container logs are written to files on the node, rotated when they reach a configured size, and pruned to a configured file count. Both defaults are modest — commonly 10 MB per file with a small number of files retained — so a verbose application can cycle through its entire retained history in minutes. kubectl logs reads only what remains, which is why a request for the last hour of output can return the last ninety seconds. This is not a bug and not a kubectl limitation: the data is gone from the node.

Most common causes

• A verbose application exceeding the rotation size quickly.
• Log rotation settings retaining too little for the application's volume.
• The container having restarted, so older output belongs to a previous instance.
• The pod having moved to a different node, leaving the old logs behind.
• Node disk pressure triggering more aggressive cleanup.
• A logging library buffering output that was never flushed before the process died.

How to diagnose it

1. Check how far back the logs go: kubectl logs POD --timestamps | head -1.
2. Check the container's restart count, since a restart divides the history.
3. Check the kubelet's log rotation settings on the node.
4. Look at the log files directly under /var/log/pods on the node to see what is retained.
5. Check whether a log aggregation system has the missing period.

How to fix it

1. Ship logs to an aggregation system. Node-local logs are a debugging convenience, not a record.
2. Increase the kubelet's log rotation size and file count if slightly more local retention is genuinely useful.
3. Reduce log verbosity in production so the retained window covers a useful period.
4. Disable output buffering in the application so nothing is lost when a process dies abruptly.
5. Capture logs immediately during an incident rather than relying on being able to return to them.

Notes

Increasing local retention trades disk against history, and node disk pressure evicts pods. Raising the limits substantially on a busy node can cause a different and worse problem than the one it solves, which is the argument for shipping logs elsewhere instead.

Related

• previous terminated container not found — There is no earlier container to read logs from
• DiskPressure — The node is low on disk or inodes

Sources

• Kubernetes documentation — Logging Architecture
• Kubernetes documentation — kubectl logs
• Kubernetes documentation — Kubelet Configuration reference