56.3 Troubleshooting order and exit codes: a playbook for broken containers
Friends, containers break. The app crashes, the port does not answer, the disk fills, the memory runs out. Beginners panic and try random commands. Experts follow the same order every time: ps, logs, inspect, exec, events, stats. And they read the exit code like a doctor reads a pulse. Today, step by step, we break containers on purpose and fix each one with this playbook.
मित्रांनो, containers बिघडतात. App crash होतं, port उत्तर देत नाही, disk भरते, memory संपते. नवशिके घाबरून कोणतेही commands चालवतात. तज्ज्ञ प्रत्येक वेळी एकच क्रम पाळतात: ps, logs, inspect, exec, events, stats. आणि doctor नाडी वाचतो तसा exit code वाचतात. आज step by step आपण containers मुद्दाम बिघडवू आणि या playbook ने प्रत्येक दुरुस्त करू.
दोस्तों, containers बिगड़ते हैं. App crash होता है, port जवाब नहीं देता, disk भर जाती है, memory खत्म हो जाती है. नए लोग घबराकर कोई भी command चलाते हैं. Experts हर बार एक ही क्रम मानते हैं: ps, logs, inspect, exec, events, stats. और doctor जैसे नब्ज़ पढ़ता है वैसे exit code पढ़ते हैं. आज step by step हम containers जानबूझकर बिगाड़ेंगे और इस playbook से हर एक ठीक करेंगे.
1. Start from a story
At Flipkart, during a sale, an on-call engineer named Neha got an alert: the cart service is down. Her first instinct was to restart everything.
Her senior stopped her and opened a notebook with six lines: ps, logs, inspect, exec, events, stats. docker ps showed the container Restarting. docker logs showed nothing useful. docker inspect showed exit code 137 and OOMKilled true.
The container was being killed for using more memory than its limit. A new feature loaded the whole product list into memory. They raised the limit for the sale and fixed the code the next day. The whole diagnosis took three minutes.
Today we learn that notebook, line by line.
2. The six steps
Step 1: Step 1: docker ps -a
Why: Is the container even running?
What: Always start by listing all containers, including stopped ones. The STATUS column already tells you a lot: Up, Exited with a code, Restarting, or unhealthy.
Example 1: List all
Type this:
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
You will see:
NAMES STATUS PORTS
cart Restarting (137) 5s ago
web Up 2 hours 0.0.0.0:8080->80/tcp
What it means: cart is in a restart loop with code 137.
Example 2: Only problems
Type this:
docker ps -a --filter status=exited --filter status=restarting --format '{{.Names}} {{.Status}}'
You will see:
cart Restarting (137) 3 seconds ago
What it means: Filter out the healthy ones.
Example 3: Unhealthy ones
Type this:
docker ps --filter health=unhealthy --format '{{.Names}}'
You will see:
api
What it means: Running but failing its health check.
Step 2: Step 2: docker logs
Why: What did the app print?
What: Most crashes explain themselves in the last lines of the log. Use --tail to see the end and --timestamps to match times with the alert.
Example 4: Last lines
Type this:
docker logs --tail 20 api
You will see:
Error: connect ECONNREFUSED 172.18.0.5:5432
What it means: The app cannot reach the database.
Example 5: With time
Type this:
docker logs --since 10m --timestamps api | tail -3
You will see:
2026-10-25T06:40:01Z Error: connect ECONNREFUSED ...
What it means: When it started failing.
Example 6: Previous run
Type this:
docker logs api 2>&1 | grep -i -m3 error
You will see:
Error: DB_HOST is not set
What it means: Logs survive restarts, so you see why the old run died.
Step 3: Step 3: docker inspect
Why: The facts Docker knows.
What: inspect shows the exit code, whether the kernel killed it for memory, restart count, environment, mounts and networks. Use --format to pick one field.
Example 7: State
Type this:
docker inspect --format '{{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} restarts={{.RestartCount}}' cart
You will see:
restarting exit=137 oom=true restarts=14
What it means: Killed for memory 14 times.
Example 8: Environment
Type this:
docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' api | grep DB_
You will see:
DB_PORT=5432
What it means: DB_HOST is missing. That is the bug.
Example 9: Mounts
Type this:
docker inspect --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}' web
You will see:
/srv/site -> /usr/share/nginx/html
What it means: Check the host folder really has files.
Step 4: Step 4: docker exec
Why: Look inside a running container.
What: If the container is running, open a shell and test from inside: is the file there, does DNS work, can I reach the database port.
Example 10: Open a shell
Type this:
docker exec -it web sh -c 'ls /usr/share/nginx/html | head -3'
You will see:
index.html
style.css
What it means: The files are there.
Example 11: DNS from inside
Type this:
docker exec api getent hosts db
You will see:
172.18.0.5 db
What it means: The name resolves on the shared network.
Example 12: Port from inside
Type this:
docker exec api sh -c 'nc -zv db 5432'
You will see:
db (172.18.0.5:5432) open
What it means: The database answers; the problem is elsewhere.
Step 5: Step 5: docker events
Why: What happened, in order.
What: events streams everything Docker does: start, die, kill, oom, health_status. Looking at the last few minutes shows the sequence of a crash.
Example 13: Last 10 minutes
Type this:
docker events --since 10m --until 0s --filter container=cart --format '{{.Time}} {{.Action}}'
You will see:
1761371001 oom
1761371001 die
1761371002 start
What it means: oom then die then restart, again and again.
Example 14: Only die events
Type this:
docker events --since 1h --until 0s --filter event=die --format '{{.Actor.Attributes.name}} exit={{.Actor.Attributes.exitCode}}'
You will see:
cart exit=137
worker exit=1
What it means: Every crash with its code.
Example 15: Health changes
Type this:
docker events --since 30m --until 0s --filter event=health_status --format '{{.Actor.Attributes.name}} {{.Action}}'
You will see:
api health_status: unhealthy
What it means: When the health check started failing.
Step 6: Step 6: docker stats
Why: How much is it using?
What: stats shows live CPU, memory against limit, network and disk input and output. A container close to its memory limit is about to be OOM killed.
Example 16: One snapshot
Type this:
docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
You will see:
NAME CPU % MEM USAGE / LIMIT
cart 98.2% 255.8MiB / 256MiB
What it means: Memory is at its limit.
Example 17: Raise the limit
Type this:
docker update --memory 512m --memory-swap 512m cart && docker inspect --format '{{.HostConfig.Memory}}' cart
You will see:
cart
536870912
What it means: 512 MB now, without recreating.
Example 18: Check again
Type this:
docker stats --no-stream --format '{{.Name}} {{.MemPerc}}' cart
You will see:
cart 61.40%
What it means: Stable. Now fix the code.
3. Read the exit code step by step
Step 7: Exit 0 and exit 1
Why: Normal end and app error.
What: Exit code 0 means the main process finished on purpose. A web server should never exit 0, so check that CMD does not run something that ends. Exit code 1 is a general application error; read the logs.
Example 19: Exit 0
Type this:
docker run --name once alpine echo hi ; docker inspect --format '{{.State.ExitCode}}' once
You will see:
hi
0
What it means: The command finished; the container stops. Normal.
Example 20: Exit 1
Type this:
docker run --name bad alpine sh -c 'echo boom; exit 1' ; docker inspect --format '{{.State.ExitCode}}' bad
You will see:
boom
1
What it means: The app reported an error.
Example 21: Clean up
Type this:
docker rm once bad
You will see:
once
bad
What it means: Remove test containers.
Step 8: Exit 125, 126 and 127
Why: Docker or command problems.
What: 125 means docker run itself failed, for example a bad flag. 126 means the command exists but cannot be executed. 127 means the command was not found inside the image.
Example 22: 125
Type this:
docker run --bad-flag alpine ; echo exit=$?
You will see:
unknown flag: --bad-flag
exit=125
What it means: Docker never started the container.
Example 23: 126
Type this:
docker run --rm alpine /etc/hostname ; echo exit=$?
You will see:
permission denied ... exit=126
What it means: The file is not executable.
Example 24: 127
Type this:
docker run --rm alpine bash ; echo exit=$?
You will see:
exec: "bash": executable file not found in $PATH
exit=127
What it means: Alpine has sh, not bash.
Step 9: Exit 137 and 143
Why: Killed from outside.
What: 137 is 128 plus 9, the KILL signal. It means out of memory or docker kill or a stop that timed out. 143 is 128 plus 15, the TERM signal: a clean docker stop.
Example 25: OOM gives 137
Type this:
docker run --name hog --memory 32m alpine sh -c 'tail /dev/zero' ; docker inspect --format '{{.State.ExitCode}} {{.State.OOMKilled}}' hog
You will see:
137 true
What it means: Killed for memory.
Example 26: Stop gives 143
Type this:
docker run -d --name s nginx:alpine >/dev/null && docker stop s >/dev/null && docker inspect --format '{{.State.ExitCode}}' s
You will see:
0
What it means: nginx handles TERM and exits 0. Apps that ignore it give 143 or 137 after 10 seconds.
Example 27: Clean up
Type this:
docker rm hog s
You will see:
hog
s
What it means: Done.
4. Common broken cases
Step 10: Port works inside but not outside
Why: The most common network bug.
What: If curl works inside the container but not from the host, either the port is not published or the app listens on 127.0.0.1 inside the container instead of 0.0.0.0.
Example 28: Check published ports
Type this:
docker port api
You will see:
(no output)
What it means: Nothing published. Recreate with -p 3000:3000.
Example 29: Check listen address
Type this:
docker exec api sh -c 'netstat -tln | grep 3000'
You will see:
tcp 0 0 127.0.0.1:3000 0.0.0.0:* LISTEN
What it means: Listening only on localhost inside. Set HOST=0.0.0.0.
Example 30: After the fix
Type this:
curl -s localhost:3000/health
You will see:
{"status":"ok"}
What it means: Reachable from the host.
Step 11: The container exits right away
Why: Nothing keeps it alive.
What: A container lives as long as its main process. If the CMD finishes or crashes at start, the container stops. Run it in the foreground and override the command to explore.
Example 31: See it exit
Type this:
docker run -d --name w myapp:1.0 && sleep 2 && docker ps -a --filter name=w --format '{{.Status}}'
You will see:
Exited (1) 1 second ago
What it means: Crashed at start.
Example 32: Run in the foreground
Type this:
docker run --rm myapp:1.0
You will see:
Error: Cannot find module '/app/server.js'
What it means: The file path in CMD is wrong.
Example 33: Explore with a shell
Type this:
docker run --rm -it --entrypoint sh myapp:1.0 -c 'ls /app'
You will see:
dist package.json
What it means: The file is in /app/dist. Fix CMD.
Step 12: The image will not pull or the disk is full
Why: Problems before the app even starts.
What: Two more failures stop a container before your code runs. A pull can fail because the name or tag is wrong or you are not logged in to a private registry. A run can fail because the Docker disk is full. Check both with small commands before you blame the app.
Example 34: Wrong tag
Type this:
docker pull nginx:1.99 ; echo exit=$?
You will see:
Error response from daemon: manifest for nginx:1.99 not found: manifest unknown
exit=1
What it means: The tag does not exist. List real tags in the registry and fix the name.
Example 35: Not logged in
Type this:
docker pull 123456789012.dkr.ecr.ap-south-1.amazonaws.com/shop-api:a1b2c3d
You will see:
no basic auth credentials
What it means: Log in first with aws ecr get-login-password piped to docker login --password-stdin.
Example 36: Disk full
Type this:
docker system df && df -h /var/lib/docker | tail -1
You will see:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 48 6 21.3GB 17.9GB (84%)
/dev/xvda1 30G 30G 0 100% /
What it means: No space left. Use the safe prune commands from the disk cleanup lesson.
5. Common mistakes and how Neha avoids them
Mistake one: restarting before looking. A restart wipes the evidence in memory. Neha reads ps, logs and inspect first.
Mistake two: using docker rm on a crashed container. The logs and exit code go with it. She copies them first.
Mistake three: guessing at OOM. She checks OOMKilled in inspect instead of guessing.
Mistake four: skipping events. The order of events often tells the whole story.
Check yourself
Q1. What is the troubleshooting order?
ps, logs, inspect, exec, events, stats.
Q2. What does exit code 137 mean?
Killed with signal 9, often out of memory; check OOMKilled.
Q3. What does exit code 127 mean?
The command was not found inside the image.
Q4. Port works inside but not from the host?
Not published, or the app listens on 127.0.0.1 instead of 0.0.0.0.
Friends, never restart first. Follow ps, logs, inspect, exec, events and stats, and read the exit code before you touch anything. Did you understand? Let us try it once more. Break three containers on purpose, one with a memory limit, one with a wrong command, one with a wrong listen address, and fix each using only the playbook.
मित्रांनो, आधी कधीही restart करू नका. ps, logs, inspect, exec, events आणि stats हा क्रम पाळा, आणि काहीही बदलण्याआधी exit code वाचा. तीन containers मुद्दाम बिघडवा, एक memory limit ने, एक चुकीच्या command ने, एक चुकीच्या listen address ने, आणि फक्त playbook वापरून प्रत्येक दुरुस्त करा.
दोस्तों, पहले कभी restart मत कीजिए. ps, logs, inspect, exec, events और stats का क्रम मानिए, और कुछ भी बदलने से पहले exit code पढ़िए. तीन containers जानबूझकर बिगाड़िए, एक memory limit से, एक गलत command से, एक गलत listen address से, और सिर्फ playbook से हर एक ठीक कीजिए.
Ravindra Bagale's Tip
Write the six steps on a sticky note next to your screen. Under pressure, a checklist beats memory every time.
हे सहा steps एका sticky note वर लिहून screen शेजारी लावा. दबावाखाली checklist नेहमी स्मरणशक्तीपेक्षा चांगली ठरते.
ये छह steps एक sticky note पर लिखकर screen के पास लगाइए. दबाव में checklist हमेशा याददाश्त से बेहतर होती है.
Lab: try at home
- Run docker ps -a and explain every STATUS you see.
- Cause an OOM kill and confirm it with inspect.
- Cause exit codes 125, 126 and 127.
- Find a missing environment variable using inspect.
- Fix an app that listens on 127.0.0.1 inside the container.
- Read the die events of the last hour.
- Raise a memory limit with docker update.
Recap
- Order: ps, logs, inspect, exec, events, stats.
- Exit 0 finished, 1 app error, 125 to 127 docker or command problems.
- 137 is KILL, often OOM; 143 is TERM.
- Port bugs: not published or wrong listen address.
- Never restart or remove before reading the evidence.