Ravindra BagaleCourses & study guides Track your progress

Guides

OOMKilled in Kubernetes and Docker: What It Means and How to Fix It

OOMKilled means your container used more memory than its limit, so the Linux kernel killed it. You see Reason: OOMKilled and Exit Code: 137 in kubectl describe pod, or "OOMKilled": true in docker inspect. The fix is to measure real memory use, set an honest limit with some headroom, and tell the runtime (Java, Node.js, Python workers) how much memory it really has.

Quick answer

kubectl describe pod api-7f9c-zx81 | grep -A4 "Last State"   # Reason: OOMKilled, Exit Code: 137
kubectl top pod api-7f9c-zx81                                # real memory use now
docker inspect --format '{{.State.OOMKilled}}' api           # Docker: true / false

A live example first

We run a small Python script in Docker with a 100 MB limit. The script builds a big list in memory:

docker run --name memtest --memory=100m python:3.12-slim \
  python -c "x = ['A' * 1024 * 1024 for _ in range(300)]"
echo $?
137
docker inspect --format '{{.State.OOMKilled}}' memtest
true

The script tried to hold about 300 MB in a 100 MB box. The kernel stopped it, and the exit code was 137 (128 + signal 9). Kubernetes shows the same thing:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Restart Count:  6

When this keeps happening, the pod goes into CrashLoopBackOff.

Why does the kernel kill the container?

OOMKilled: memory above the limit Memory use grows past the container limit, the kernel kills the process with exit code 137, then you measure and fix. OOMKilled: memory above the limit Memory growsusage upLimit hite.g. 512MiKernel killexit 137Measurekubectl topTunelimit + runtime you Fix it with the right limit and runtime settings, not by removing the limit

OOMKilled = the container went above its memory limit. Measure first, then tune the limit and the runtime.

A memory limit is a hard wall, set through Linux cgroups. CPU limits only slow a container down, but memory cannot be slowed down. When the container goes over its limit, the kernel's OOM killer ends the biggest process inside it. Kubernetes did not "decide" to kill your app. It only reports what the kernel did.

How do I fix OOMKilled step by step?

  1. Confirm the reason is really OOMKilled (exit 137 without OOM can be a liveness probe).
  2. Run kubectl top pod during normal traffic for at least 10 minutes.
  3. Note the peak memory.
  4. Set requests.memory close to normal use.
  5. Set limits.memory about 25 to 50 percent above the peak.
  6. Tune the runtime for your language (sections below).
  7. Apply, and watch kubectl top pod and the restart count for a day.
  8. If memory keeps climbing, treat it as a leak and profile the app.
resources:
  requests:
    memory: "384Mi"
  limits:
    memory: "512Mi"

Fix it in your language

Java / Spring Boot

Modern JVMs (Java 11+, and 8u191+) read the container limit. Tell the JVM to use a percentage of it, not a fixed -Xmx larger than the limit:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=70.0"

Leave about 30 percent for metaspace, threads and buffers. A 512Mi limit with -Xmx512m will always be OOMKilled.

Node.js

The V8 heap has its own size limit. Set it below the container limit:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=384"

For a 512Mi limit, 384 MB of heap leaves room for buffers and native code.

Python (Gunicorn / Celery)

Every Gunicorn worker is a full copy of the app. 4 workers × 150 MB = 600 MB, which does not fit in 512Mi. Lower --workers, or raise the limit. For Celery, --max-tasks-per-child=100 restarts workers before slow leaks grow.

Go

Go 1.19+ supports a soft memory limit, so the garbage collector works harder before the hard limit:

env:
  - name: GOMEMLIMIT
    value: "450MiB"

.NET

.NET reads container limits too. You can cap the GC heap:

env:
  - name: DOTNET_GCHeapHardLimitPercent
    value: "0x46"    # 70 percent, in hex

Databases: MySQL, PostgreSQL, Redis, MongoDB

  • MySQL: innodb_buffer_pool_size must fit inside the limit (about 60 to 70 percent).
  • PostgreSQL: shared_buffers plus work_mem × connections must fit.
  • Redis: set maxmemory and an eviction policy such as allkeys-lru.
  • MongoDB: set --wiredTigerCacheSizeGB lower than the default, which assumes a whole machine.

OOMKilled vs eviction

OOMKilled Evicted
Who acts Linux kernel kubelet
Why One container passed its own limit The whole node is low on memory
What you see Reason: OOMKilled, exit 137 Pod status Evicted, message "The node was low on resource: memory"
Fix Limit and runtime settings Set requests, add nodes, fix noisy neighbours

Real example

Discord published an engineering blog post in 2020 about its "Read States" service. The Go version showed regular latency spikes from garbage collection, and the team rewrote it in Rust to control memory use directly. Most of us will not rewrite a service. The takeaway is that memory behaviour is part of the design, so measure it like any other number. Go's later GOMEMLIMIT setting exists to help with exactly this kind of problem.

Mistakes students make (and how to avoid them)

  1. Raising the limit every time. If memory climbs steadily under the same traffic, a bigger limit only delays the next kill. Find the leak with a heap dump (Java), --inspect heap snapshots (Node.js) or tracemalloc (Python).
  2. Setting the limit far above the request. The scheduler places pods by request. If many pods burst to their limits together, the node runs out and pods are evicted.
  3. Hard-coding -Xmx from a VM. A setting that was fine on an 8 GB VM is wrong in a 512Mi container. Use percentage-based settings.
  4. Watching only averages. Memory spikes at startup, during big reports or nightly jobs. Check the peak over a full day in CloudWatch Container Insights, Prometheus or Grafana.
  5. Ignoring the node. Run kubectl describe node and check MemoryPressure. If it is True, you have an eviction problem, not only an OOMKilled one.

Practice lab (15 minutes)

  1. Run the Python memtest example with --memory=100m and confirm exit 137.
  2. Run it again with --memory=400m and confirm it finishes.
  3. In Kubernetes, set a 128Mi limit on a test pod, trigger the kill, and read Last State.

Summary table

Stack Setting Example for 512Mi limit
Java -​XX:​MaxRAMPercentage 70.0
Node.js --​max-​old-​space-​size 384
Python Gunicorn --workers 2 or 3
Go GOMEMLIMIT 450MiB
.NET DOTNET_​GCHeapHardLimitPercent 0x46 (70%)
MySQL innodb_​buffer_​pool_​size 320M
Redis maxmemory 400mb

Quick quiz: check what you learned

  1. What does exit code 137 with Reason OOMKilled mean?

    1. The memory limit was exceeded
    2. The image could not be pulled
    3. The CPU request was too large
    4. The liveness probe path is bad
    Show answer

    A. The kernel killed the process with signal 9 because it went over its memory limit.

  2. Which command shows live memory use of pods?

    1. kubectl top pod
    2. kubectl get pvc
    3. kubectl get ingress
    4. kubectl cordon node
    Show answer

    A. kubectl top uses metrics-server to show current CPU and memory.

  3. Why leave headroom above the Java heap size?

    1. The JVM uses non-heap memory
    2. Kubernetes adds an extra limit
    3. The image layers use the RAM
    4. Probes reserve half the memory
    Show answer

    A. Metaspace, threads and buffers live outside the heap but count against the limit.

  4. Memory grows every hour until OOMKilled. What is likely?

    1. A memory leak in the app
    2. A wrong image tag in YAML
    3. A missing toleration rule
    4. A Docker Hub rate limit
    Show answer

    A. Steady growth under steady load points to a leak, so profile the app.

Learn it properly

This guide is the short path. The free DevOps course covers Kubernetes step by step, with labs:

Frequently asked questions

What does OOMKilled mean?

Out Of Memory Killed. The container used more memory than its limit, so the Linux kernel killed the process with signal 9.

Why is the exit code 137?

137 is 128 + 9, which means the process was ended by signal 9 (SIGKILL). OOM is the most common reason.

How do I check OOMKilled in Docker?

Run docker inspect --format "{{.State.OOMKilled}}" CONTAINER. It prints true if the kernel killed it for memory.

Should I just remove the memory limit?

No. Without a limit, one leaking pod can use the whole node and cause other pods to be evicted.

Why does my Java app get OOMKilled with a small heap?

The JVM also uses memory outside the heap: metaspace, threads and buffers. Leave 25 to 30 percent of the limit for them.

Is OOMKilled the same as eviction?

No. OOMKilled is one container hitting its own limit. Eviction is the kubelet removing pods because the whole node is low on memory.