OOMKilled in Kubernetes and Docker: What It Means and How to Fix It
OOMKilled means your container used more memory than its limit, so the Linux kernel killed it. You see Reason: OOMKilled and Exit Code: 137 in kubectl describe pod, or "OOMKilled": true in docker inspect. The fix is to measure real memory use, set an honest limit with some headroom, and tell the runtime (Java, Node.js, Python workers) how much memory it really has.
Quick answer
kubectl describe pod api-7f9c-zx81 | grep -A4 "Last State" # Reason: OOMKilled, Exit Code: 137
kubectl top pod api-7f9c-zx81 # real memory use now
docker inspect --format '{{.State.OOMKilled}}' api # Docker: true / false
A live example first
We run a small Python script in Docker with a 100 MB limit. The script builds a big list in memory:
docker run --name memtest --memory=100m python:3.12-slim \
python -c "x = ['A' * 1024 * 1024 for _ in range(300)]"
echo $?
137
docker inspect --format '{{.State.OOMKilled}}' memtest
true
The script tried to hold about 300 MB in a 100 MB box. The kernel stopped it, and the exit code was 137 (128 + signal 9). Kubernetes shows the same thing:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Restart Count: 6
When this keeps happening, the pod goes into CrashLoopBackOff.
Why does the kernel kill the container?
OOMKilled = the container went above its memory limit. Measure first, then tune the limit and the runtime.
A memory limit is a hard wall, set through Linux cgroups. CPU limits only slow a container down, but memory cannot be slowed down. When the container goes over its limit, the kernel's OOM killer ends the biggest process inside it. Kubernetes did not "decide" to kill your app. It only reports what the kernel did.
How do I fix OOMKilled step by step?
- Confirm the reason is really
OOMKilled(exit 137 without OOM can be a liveness probe). - Run
kubectl top podduring normal traffic for at least 10 minutes. - Note the peak memory.
- Set
requests.memoryclose to normal use. - Set
limits.memoryabout 25 to 50 percent above the peak. - Tune the runtime for your language (sections below).
- Apply, and watch
kubectl top podand the restart count for a day. - If memory keeps climbing, treat it as a leak and profile the app.
resources:
requests:
memory: "384Mi"
limits:
memory: "512Mi"
Fix it in your language
Java / Spring Boot
Modern JVMs (Java 11+, and 8u191+) read the container limit. Tell the JVM to use a percentage of it, not a fixed -Xmx larger than the limit:
env:
- name: JAVA_TOOL_OPTIONS
value: "-XX:MaxRAMPercentage=70.0"
Leave about 30 percent for metaspace, threads and buffers. A 512Mi limit with -Xmx512m will always be OOMKilled.
Node.js
The V8 heap has its own size limit. Set it below the container limit:
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=384"
For a 512Mi limit, 384 MB of heap leaves room for buffers and native code.
Python (Gunicorn / Celery)
Every Gunicorn worker is a full copy of the app. 4 workers × 150 MB = 600 MB, which does not fit in 512Mi. Lower --workers, or raise the limit. For Celery, --max-tasks-per-child=100 restarts workers before slow leaks grow.
Go
Go 1.19+ supports a soft memory limit, so the garbage collector works harder before the hard limit:
env:
- name: GOMEMLIMIT
value: "450MiB"
.NET
.NET reads container limits too. You can cap the GC heap:
env:
- name: DOTNET_GCHeapHardLimitPercent
value: "0x46" # 70 percent, in hex
Databases: MySQL, PostgreSQL, Redis, MongoDB
- MySQL:
innodb_buffer_pool_sizemust fit inside the limit (about 60 to 70 percent). - PostgreSQL:
shared_bufferspluswork_mem× connections must fit. - Redis: set
maxmemoryand an eviction policy such asallkeys-lru. - MongoDB: set
--wiredTigerCacheSizeGBlower than the default, which assumes a whole machine.
OOMKilled vs eviction
| OOMKilled | Evicted | |
|---|---|---|
| Who acts | Linux kernel | kubelet |
| Why | One container passed its own limit | The whole node is low on memory |
| What you see | Reason: OOMKilled, exit 137 |
Pod status Evicted, message "The node was low on resource: memory" |
| Fix | Limit and runtime settings | Set requests, add nodes, fix noisy neighbours |
Real example
Discord published an engineering blog post in 2020 about its "Read States" service. The Go version showed regular latency spikes from garbage collection, and the team rewrote it in Rust to control memory use directly. Most of us will not rewrite a service. The takeaway is that memory behaviour is part of the design, so measure it like any other number. Go's later GOMEMLIMIT setting exists to help with exactly this kind of problem.
Mistakes students make (and how to avoid them)
- Raising the limit every time. If memory climbs steadily under the same traffic, a bigger limit only delays the next kill. Find the leak with a heap dump (Java),
--inspectheap snapshots (Node.js) ortracemalloc(Python). - Setting the limit far above the request. The scheduler places pods by request. If many pods burst to their limits together, the node runs out and pods are evicted.
- Hard-coding
-Xmxfrom a VM. A setting that was fine on an 8 GB VM is wrong in a 512Mi container. Use percentage-based settings. - Watching only averages. Memory spikes at startup, during big reports or nightly jobs. Check the peak over a full day in CloudWatch Container Insights, Prometheus or Grafana.
- Ignoring the node. Run
kubectl describe nodeand checkMemoryPressure. If it is True, you have an eviction problem, not only an OOMKilled one.
Practice lab (15 minutes)
- Run the Python
memtestexample with--memory=100mand confirm exit 137. - Run it again with
--memory=400mand confirm it finishes. - In Kubernetes, set a 128Mi limit on a test pod, trigger the kill, and read
Last State.
Summary table
| Stack | Setting | Example for 512Mi limit |
|---|---|---|
| Java | -XX:MaxRAMPercentage |
70.0 |
| Node.js | --max-old-space-size |
384 |
| Python Gunicorn | --workers |
2 or 3 |
| Go | GOMEMLIMIT |
450MiB |
| .NET | DOTNET_GCHeapHardLimitPercent |
0x46 (70%) |
| MySQL | innodb_buffer_pool_size |
320M |
| Redis | maxmemory |
400mb |
Quick quiz: check what you learned
What does exit code 137 with Reason OOMKilled mean?
- The memory limit was exceeded
- The image could not be pulled
- The CPU request was too large
- The liveness probe path is bad
Show answer
A. The kernel killed the process with signal 9 because it went over its memory limit.
Which command shows live memory use of pods?
- kubectl top pod
- kubectl get pvc
- kubectl get ingress
- kubectl cordon node
Show answer
A. kubectl top uses metrics-server to show current CPU and memory.
Why leave headroom above the Java heap size?
- The JVM uses non-heap memory
- Kubernetes adds an extra limit
- The image layers use the RAM
- Probes reserve half the memory
Show answer
A. Metaspace, threads and buffers live outside the heap but count against the limit.
Memory grows every hour until OOMKilled. What is likely?
- A memory leak in the app
- A wrong image tag in YAML
- A missing toleration rule
- A Docker Hub rate limit
Show answer
A. Steady growth under steady load points to a leak, so profile the app.
Learn it properly
This guide is the short path. The free DevOps course covers Kubernetes step by step, with labs:
Frequently asked questions
What does OOMKilled mean?
Out Of Memory Killed. The container used more memory than its limit, so the Linux kernel killed the process with signal 9.
Why is the exit code 137?
137 is 128 + 9, which means the process was ended by signal 9 (SIGKILL). OOM is the most common reason.
How do I check OOMKilled in Docker?
Run docker inspect --format "{{.State.OOMKilled}}" CONTAINER. It prints true if the kernel killed it for memory.
Should I just remove the memory limit?
No. Without a limit, one leaking pod can use the whole node and cause other pods to be evicted.
Why does my Java app get OOMKilled with a small heap?
The JVM also uses memory outside the heap: metaspace, threads and buffers. Leave 25 to 30 percent of the limit for them.
Is OOMKilled the same as eviction?
No. OOMKilled is one container hitting its own limit. Eviction is the kubelet removing pods because the whole node is low on memory.