Kubernetes OOMKilled means the kernel killed a process in your container because memory ran out. Either the container went past its own memory limit, or the node ran out of memory altogether, which hits containers with no limit of their own too. The pod status then shows Last State: Terminated, Reason: OOMKilled and Exit Code: 137. You spot it with kubectl describe pod, and kubectl top pod shows current usage. Fixing it usually means one of three things: adjusting the values to the measured need, finding a memory leak, or telling the runtime what the container limit is. Doubling the limit is a bandage, not a fix.
I've been running Kubernetes clusters since 2017, today mostly k3s on Hetzner Cloud with Prometheus, Grafana and Loki, and I'm a CNCF Kubestronaut. OOMKilled is one of those messages that keeps coming back even though the code hasn't changed in weeks. Everything I write about Kubernetes is on the Kubernetes page.
What OOMKilled means and what exit code 137 tells you
OOM stands for "out of memory." A container with a memory limit runs inside a cgroup that the Linux kernel watches. When usage goes past the limit and the kernel comes under memory pressure, the OOM killer ends the process with SIGKILL. Exit code 137 follows the Unix convention of 128 plus the signal number, 9 being SIGKILL. The container gets no chance to clean up: no closing connections, no buffers flushed, no shutdown hook.
The second path is easy to miss. If the node runs out of memory before the kubelet can reclaim any, the same OOM killer strikes one level up, and then it hits containers with no memory limit at all. The Kubernetes documentation on node-pressure eviction (as of September 2026) covers this in the section on node out of memory behavior. The pod status reads Reason: OOMKilled either way; the node's events tell you which case it was.
The Kubernetes documentation on resource management (as of September 2026) is careful here: "memory limits are enforced reactively. A container may use more memory than its memory limit, but if it does, it may get killed." Once the kernel notices pressure, it's over. CPU is different: a CPU limit is enforced by throttling, the process slows down but is never killed.
If the container is allowed to restart, the kubelet restarts it after the OOM kill, immediately the first time and with a growing delay after that. Several times in a row and the pod ends up in CrashLoopBackOff: Debugging Pods, so OOMKilled is often just the first link in a chain.
Detecting OOMKilled in Kubernetes: three kubectl commands
Start with restarts. kubectl get pods -n <namespace> shows in the RESTARTS column which pod restarts suspiciously often. The second command gives you the reason. kubectl describe pod shows the container's last state under Containers:
kubectl describe pod api-7d9c6b5f8-x2k4q -n shop
The part that matters looks like this (shortened):
Containers:
api:
State: Running
Started: Wed, 19 Nov 2026 09:14:02 +0100
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Wed, 19 Nov 2026 09:02:41 +0100
Finished: Wed, 19 Nov 2026 09:13:58 +0100
Ready: True
Restart Count: 4
Limits:
memory: 256Mi
Requests:
cpu: 100m
memory: 256Mi
Three things to read off: the reason (OOMKilled), the exit code (137), and the limit it died at (256Mi). The timestamps tell you how long it ran. Eleven minutes, as here, points to growing usage rather than a startup failure. More everyday commands are in kubectl Commands: The Essentials.
The third command shows current usage. kubectl top needs a running metrics-server in the cluster; the Kubernetes guide on assigning memory resources (as of September 2026) lists it as a prerequisite. k3s ships it by default.
kubectl top pod -n shop --containers
POD NAME CPU(cores) MEMORY(bytes)
api-7d9c6b5f8-x2k4q api 38m 241Mi
241Mi against a 256Mi limit: this container is about to be killed again. But kubectl top is a snapshot. Whether usage is stably high or climbing shows only in a time series.
Six causes of OOMKilled and how to fix each
In practice the causes boil down to six patterns. The table maps each cause to its symptom, the check, and the fix:
The "runtime doesn't know the limit" case is the sneakiest, because the application does nothing wrong: it asks the operating system how much memory is available, gets the whole node, and sizes its heap to match, while the kernel only sees the cgroup.
Today that is the exception, not the rule. Current runtimes detect cgroup limits on their own: for the JVM, UseContainerSupport is on by default on Linux according to the Oracle documentation for the java command (as of September 2026). Three trip hazards remain: old runtime versions, images where detection is turned off (-XX:-UseContainerSupport), and a hard-wired heap size above the container limit. Check version and settings before you touch the limit. A heap ceiling you set yourself needs a gap below the container limit for threads and buffers.
Setting requests and limits properly
A pod without a memory limit can't be OOMKilled by its own limit. It can, however, starve the whole node, and then the OOM killer strikes at node level, against the neighbours and against the pod itself. That's why requests and limits belong in every production deployment. I set requests almost everywhere in my clusters, because that is what the scheduler plans with: without one it packs the node too full. A clean block looks like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: shop
spec:
replicas: 2
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: registry.example.com/shop/api:2.4.1
resources:
requests:
cpu: 100m
memory: 384Mi
limits:
memory: 384Mi
Two decisions are baked in. First, memory request and memory limit are equal. The scheduler then books exactly the amount the container may use at most. That is not an exclusive reservation: requests are accounting for scheduling, not a physically fenced-off block of memory, and memory pressure on the node stays possible. The gain lies elsewhere, in that a container which can't exceed its request is a less likely eviction target. Second, there's a CPU request but no CPU limit. CPU throttling makes applications slow without showing up in the pod status. I set CPU limits situationally, depending on the cluster and how critical the workload is, and hold them to be less critical than memory limits anyway: a throttled process keeps running, a process above its memory limit dies. The trade-off is in Kubernetes Requests and Limits Explained.
Requests and limits together determine the pod's QoS class. According to the documentation on Pod QoS classes (as of September 2026), a pod is Guaranteed only if every container has memory and CPU requests and limits, each request equal to its limit. Burstable has at least one request or limit but misses the Guaranteed criteria. BestEffort has nothing set. The example above is Burstable because the CPU limit is missing.
The QoS class is not the sorting criterion for eviction, though. The documentation on node-pressure eviction (as of September 2026) names a different order: usage above the requests first, then pod priority, then usage relative to the requests. QoS acts indirectly, because a Guaranteed pod normally never exceeds its requests and a BestEffort pod has none to exceed. That is why BestEffort usually goes first and Guaranteed last. Guaranteed is a lower risk, not protection from going down.
One more detail: set only a limit and no request, and Kubernetes copies the limit into the request. On small nodes like mine at Hetzner, that space disappears fast.
A case from practice: how a scanner pushed the small pods out
How closely that hangs together was shown to me by a vulnerability scanner. It started eight workers at once and took a lot of memory doing so. The neighbours were hit first: smaller pods on the same node were pushed out and ran out of memory. Shortly after, the scanner itself did.
It surfaced quickly and was fixed with adjusted request and limit values. What values the pods involved had before that, I did not document, so I will not reconstruct it here. The mechanism is the one from the section above: who goes first under memory pressure is decided by how far a pod sits above its requests.
The lesson, as I put it: setting request and limit far apart gets more utilization out of your nodes, because the scheduler only plans with the request while the limit leaves room upward. It also risks exactly this displacement as soon as several pods peak together. Small clusters with tight capacity are more exposed, because that is where I define things deliberately narrowly.
Catching OOMKilled earlier: Prometheus, Grafana, Loki
kubectl top covers the acute case, not prevention. My clusters run kube-prometheus-stack, and the most useful curve here is a container's working set relative to its limit. The Prometheus query:
max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
/ on (namespace, pod, container)
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
The value sits between 0 and 1. An alert on sustained values above 0.9 buys you time before the kernel strikes. How much depends on the load profile: hours to days with slowly growing usage, seconds with strongly fluctuating load. Treat it as a warning, not a safety net. A second alert on kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} reports every kill, including the one at three in the morning that the restart has papered over by breakfast. How I set up the stack on k3s is in Kubernetes Monitoring with Prometheus.
Logs help with the why. A container killed by SIGKILL writes no farewell line, but the minutes before are telling: a request with an unusually large response, a batch job loading every record at once, a cache with no upper bound. Those lines are not lost. According to the documentation on logging architecture (as of September 2026), the kubelet keeps one terminated container with its logs, and kubectl logs --previous pulls them up. It stays at that one, and once the pod leaves the node the logs go with it. For retention and correlation with metrics I collect centrally with Loki, described in Kubernetes Logging with Loki and Grafana.
A pattern that easily misleads is confusion with probes: a liveness probe fails because the container, under memory pressure, no longer answers in time, and the restart gets blamed on the probe instead of memory. A look at Last State settles it. More in Liveness and Readiness Probes Explained.
Frequently asked questions
Why are my pods OOMKilled and how do I catch it earlier?
A pod is OOMKilled when the kernel ends a process because memory ran out: either because the container exceeds its own memory limit, or because the node runs out of memory. You catch it earlier with a time series of memory usage relative to the limit, for example in Grafana, plus an alert at 90 percent. How much warning that buys you depends on the load profile.
What does exit code 137 mean in Kubernetes?
Exit code 137 is 128 plus 9, and 9 is the SIGKILL signal. Combined with Reason: OOMKilled it means the kernel killed the container for running out of memory. Without OOMKilled it can also be a manual kill -9 or a container that didn't stop within terminationGracePeriodSeconds.
Is raising the memory limit enough?
Only if the limit was guessed too low and usage is stable. If usage climbs linearly over hours, a higher limit just postpones the next kill and you pay for memory a leak is eating. Measure first, then decide.
What is the best way to collect Kubernetes logs?
Centrally, outside the pod. Not because the OOM kill wipes the logs straight away: the kubelet keeps one terminated container with its logs, and kubectl logs --previous shows them. They are gone once the pod leaves the node or rotation kicks in. For retention I use Loki with Grafana, stored in object storage. The important part is having logs and metrics side by side in one dashboard, otherwise you hunt for the same timestamp in two tools.
How do I monitor autoscaling behavior?
The Horizontal Pod Autoscaler records its decisions as events that kubectl describe hpa shows, and Prometheus adds the time series for replica count and utilization. For OOMKilled the key point is that every replica needs enough memory on its own: the HPA adds pods but doesn't save a single replica from being killed. More in Kubernetes Autoscaling with HPA.
What is the difference between OOMKilled and Evicted?
OOMKilled hits a single container: the kernel kills it for running out of memory, either because it exceeds its own limit or because the node is out of memory. The kubelet then restarts it if the restart policy calls for it. Evicted hits a whole pod that the kubelet clears before the node runs into memory pressure. The ranking goes by usage above the requests and by pod priority, not directly by QoS class. Evicted pods are not restarted but recreated by their controller; without one, they stay gone.
Where to go from here
First confirm with kubectl describe pod that Last State really says OOMKilled. Then pull up the memory curve of the last days and match the case to one of the six causes. Only then touch requests and limits, and only with numbers from the measurement. If the pod is stuck in a restart loop, CrashLoopBackOff: Debugging Pods helps you out; the fundamentals are in Kubernetes Requests and Limits Explained.
The full picture, with resources, probes, scaling and monitoring together, is in chapter 8 "Ready for Production" of my Kubernetes Practical Guide (Rheinwerk Computing).
