56.8 Debugging small and distroless containers with docker debug and netshoot
Friends, in the image size lesson we removed the shell, curl and every extra tool from our production images. That makes them small and safe. But now, when something breaks, docker exec sh says executable file not found. How do we look inside? Today, step by step, we learn three ways: docker debug, which brings its own toolbox; netshoot, a container full of network tools that joins the broken container's network; and the nsenter and ephemeral container ideas behind both.
मित्रांनो, image size lesson मध्ये आपण production images मधून shell, curl आणि प्रत्येक जास्तीचं tool काढून टाकलं. त्यामुळे त्या छोट्या आणि सुरक्षित झाल्या. पण आता काही बिघडलं की docker exec sh म्हणतो executable file not found. मग आत कसं बघायचं? आज step by step तीन मार्ग शिकू: docker debug, जे स्वतःचा toolbox घेऊन येतं; netshoot, network tools नी भरलेला container जो बिघडलेल्या container च्या network मध्ये जातो; आणि दोन्हीमागची nsenter आणि ephemeral container ची कल्पना.
दोस्तों, image size lesson में हमने production images से shell, curl और हर extra tool हटा दिया. इससे वे छोटी और सुरक्षित हो गईं. पर अब कुछ बिगड़ने पर docker exec sh कहता है executable file not found. तो अंदर कैसे देखें? आज step by step तीन तरीके सीखेंगे: docker debug, जो अपना toolbox लेकर आता है; netshoot, network tools से भरा container जो बिगड़े container के network में जुड़ता है; और दोनों के पीछे का nsenter और ephemeral container का विचार.
1. Start from a story
At PhonePe, a payments service ran on a distroless image of 40 MB. One afternoon it started returning timeouts when calling a partner bank API.
An engineer named Vikram tried docker exec payments sh. It failed, because there is no shell. He could not run curl, nslookup or ping inside. His first idea was to rebuild the image with tools added, but that would change the very thing he was trying to debug.
His lead showed him a better way. With netshoot sharing the container's network, he ran dig and found that the partner hostname resolved to an old IP. With docker debug he opened a shell with tools inside the same container and read the app's config file. The fix was a DNS change, and the image stayed untouched.
Today we learn Vikram's toolbox.
2. Why exec fails
Step 1: See the problem
Why: Small images have no tools.
What: Distroless and scratch images contain only your app and its libraries. That is good for security, but it means docker exec has nothing to run. First prove it, so you know which method you need.
Example 1: Run a distroless app
Type this:
docker run -d --name pay2 -p 8081:80 example/pay:distroless
You will see:
7a1c...
What it means: Our sample distroless service. The image name is an example.
Example 2: Try a shell
Type this:
docker exec -it pay2 sh
You will see:
OCI runtime exec failed: exec: "sh": executable file not found in $PATH
What it means: No shell inside.
Example 3: Try curl
Type this:
docker exec pay2 curl -s localhost/health
You will see:
exec: "curl": executable file not found in $PATH
What it means: No tools at all.
Step 2: Do not rebuild to debug
Why: Keep the evidence.
What: Rebuilding with tools changes the image and restarts the container, so the bug may disappear. Debug the running container as it is. Note what you see before changing anything.
Example 4: Record the state
Type this:
docker inspect --format '{{.State.Status}} {{.State.StartedAt}} restarts={{.RestartCount}}' pay2
You will see:
running 2026-10-25T05:12:40Z restarts=0
What it means: Write this down first.
Example 5: Save logs
Type this:
docker logs pay2 > pay2-before.log 2>&1 && wc -l pay2-before.log
You will see:
214 pay2-before.log
What it means: Logs kept outside the container.
Example 6: Image id
Type this:
docker inspect --format '{{.Image}}' pay2 | cut -c1-19
You will see:
sha256:4be9d7c1a2f3
What it means: Prove later that you debugged this exact image.
3. docker debug step by step
Step 3: What docker debug is
Why: A toolbox you bring along.
What: docker debug is part of Docker Desktop and newer Docker subscriptions. It attaches a shell with tools like curl, vim and htop to any container, even one without a shell, and the tools are not added to the image. On a plain Linux server without it, use netshoot or nsenter, shown later.
Example 7: Check it exists
Type this:
docker debug --help | head -2
You will see:
Usage: docker debug [OPTIONS] {CONTAINER|IMAGE}
Get a shell into any container or image.
What it means: The plugin is installed.
Example 8: Open a debug shell
Type this:
docker debug pay2
You will see:
docker >
What it means: A shell with tools, inside pay2's filesystem and processes.
Example 9: Look around
Type this:
ls /app && cat /app/config.yaml | head -3
You will see:
pay
config.yaml
bank_url: https://api.partner-bank.example
timeout: 3s
What it means: We can read files that exec could not show.
Step 4: Use tools inside docker debug
Why: curl, ps and more.
What: Inside the debug shell you can run curl against localhost, see processes and read the environment. The container's own process is untouched.
Example 10: Call the app
Type this:
curl -s localhost/health
You will see:
{"status":"degraded","bank":"timeout"}
What it means: The app is up but the bank call times out.
Example 11: Processes
Type this:
ps -ef | head -3
You will see:
UID PID CMD
65532 1 /app/pay
What it means: The app runs as a non-root user.
Example 12: Install a tool for this session
Type this:
install bind-tools && dig +short api.partner-bank.example
You will see:
203.0.113.10
What it means: Tools come from the toolbox, not the image.
Step 5: Debug an image, not a container
Why: Look before you run.
What: docker debug can also open a shell into an image to explore its files without starting the app. Useful to check what a vendor image contains.
Example 13: Explore an image
Type this:
docker debug example/pay:distroless
You will see:
docker >
What it means: A temporary shell over the image.
Example 14: List files
Type this:
ls -la /app
You will see:
-rwxr-xr-x 1 65532 65532 12M pay
-rw-r--r-- 1 65532 65532 220 config.yaml
What it means: Binary and config, safe modes.
Example 15: Leave
Type this:
exit
You will see:
(back to your terminal)
What it means: Nothing was changed in the image.
4. netshoot step by step
Step 6: What netshoot is
Why: A Swiss army knife for networks.
What: nicolaka/netshoot is a container image packed with dig, curl, tcpdump, iperf, nmap and many more. You run it with --network container:NAME so it shares the network of the broken container: same IP, same DNS settings, same ports.
Example 16: Join the network
Type this:
docker run --rm -it --network container:pay2 nicolaka/netshoot
You will see:
pay2 ~ #
What it means: A shell in pay2's network namespace.
Example 17: Same IP
Type this:
ip -4 addr show eth0 | grep inet
You will see:
inet 172.17.0.3/16
What it means: Exactly pay2's address.
Example 18: Same DNS config
Type this:
cat /etc/resolv.conf | head -2
You will see:
nameserver 127.0.0.11
options ndots:0
What it means: What the app really uses.
Step 7: DNS problems
Why: The most common network bug.
What: dig shows what a name resolves to from inside the container's network. Compare it with the right answer to find stale records or wrong DNS servers.
Example 19: Resolve the partner
Type this:
dig +short api.partner-bank.example
You will see:
198.51.100.7
What it means: From the container: an old IP.
Example 20: Ask a public resolver
Type this:
dig +short @1.1.1.1 api.partner-bank.example
You will see:
203.0.113.10
What it means: The real current IP is different.
Example 21: Find the cause
Type this:
dig api.partner-bank.example | grep -A1 'ANSWER SECTION'
You will see:
api.partner-bank.example. 86400 IN A 198.51.100.7
What it means: A long TTL in the internal DNS kept the old address.
Step 8: Connection and port problems
Why: Can we reach it at all?
What: curl with -v shows each step of a connection. nc tests a single port. These separate DNS problems from firewall problems.
Example 22: Verbose curl
Type this:
curl -sv --max-time 5 https://203.0.113.10/ping -o /dev/null 2>&1 | grep -E 'Connected|HTTP/'
You will see:
* Connected to 203.0.113.10 port 443
< HTTP/2 200
What it means: The new IP works.
Example 23: Port test
Type this:
nc -zv -w 3 198.51.100.7 443
You will see:
nc: connect to 198.51.100.7 port 443 (tcp) timed out
What it means: The old IP is dead. That is our timeout.
Example 24: Inside the Docker network
Type this:
nc -zv db 5432
You will see:
db (172.18.0.4:5432) open
What it means: Internal services are fine.
Step 9: Capture packets with tcpdump
Why: See the real traffic.
What: When nothing else explains it, tcpdump shows every packet. Write to a file and open it in Wireshark on your laptop. Capture only what you need and delete the file after, because packets can contain private data.
Example 25: Live capture
Type this:
tcpdump -ni eth0 -c 5 port 443
You will see:
IP 172.17.0.3.51122 > 198.51.100.7.443: Flags [S]
IP 172.17.0.3.51122 > 198.51.100.7.443: Flags [S]
What it means: Only SYN packets, no reply. The far side is gone.
Example 26: Save to a file
Type this:
tcpdump -ni eth0 -c 200 -w /tmp/pay.pcap port 443 && ls -l /tmp/pay.pcap
You will see:
-rw-r--r-- 1 root root 31244 /tmp/pay.pcap
What it means: A short capture.
Example 27: Copy it out
Type this:
docker run --rm -v "$PWD":/out --network container:pay2 nicolaka/netshoot sh -c 'tcpdump -ni eth0 -c 50 -w /out/pay.pcap port 443' && ls pay.pcap
You will see:
pay.pcap
What it means: Open in Wireshark; delete when done.
Step 10: netshoot in Docker Compose
Why: Debug a whole stack.
What: In a Compose project, add a debug service that uses the same network as the service you want to look at, behind a profile so it never starts in normal runs. Start it only when you need it and remove it after.
Example 28: The debug service
Type this:
services:
pay:
image: example/pay:distroless
debug:
image: nicolaka/netshoot
network_mode: "service:pay"
command: sleep infinity
profiles: ["debug"]
You will see:
(compose.yaml)
What it means: Shares pay's network only when the debug profile is on.
Example 29: Start only the debugger
Type this:
docker compose --profile debug up -d debug && docker compose exec debug dig +short db
You will see:
172.20.0.3
What it means: Debug tools next to the app, on demand.
Example 30: Remove it
Type this:
docker compose --profile debug rm -sf debug
You will see:
Container shop-debug-1 Removed
What it means: The normal stack is untouched.
5. Without any extra tool: nsenter
Step 11: nsenter from the host
Why: Use the host's own tools.
What: Every container is a process on the host. nsenter enters that process's network namespace, so host tools like ss and curl see the container's network. This needs root on the host.
Example 31: Find the PID
Type this:
PID=$(docker inspect --format '{{.State.Pid}}' pay2) && echo $PID
You will see:
48211
What it means: The container's main process on the host.
Example 32: See its sockets
Type this:
sudo nsenter -t $PID -n ss -tlnp
You will see:
LISTEN 0 4096 *:80 *:* users:(("pay",pid=48211))
What it means: The app listens on port 80 inside.
Example 33: Curl as the container
Type this:
sudo nsenter -t $PID -n curl -s localhost/health
You will see:
{"status":"degraded","bank":"timeout"}
What it means: Same result, no extra image needed.
Step 12: Share the process namespace
Why: See the app's processes and files.
What: With --pid container:NAME a debug container can see the app's processes and reach its files through /proc/1/root. This is the same idea as Kubernetes ephemeral containers with kubectl debug.
Example 34: Join pid and network
Type this:
docker run --rm -it --pid container:pay2 --network container:pay2 nicolaka/netshoot ps -ef | head -3
You will see:
PID USER CMD
1 65532 /app/pay
What it means: We see the app as PID 1.
Example 35: Read its files
Type this:
docker run --rm --pid container:pay2 alpine cat /proc/1/root/app/config.yaml | head -1
You will see:
bank_url: https://api.partner-bank.example
What it means: The app filesystem without a shell in it.
Example 36: Kubernetes equivalent
Type this:
kubectl debug -it pay-7c9-x2 --image=nicolaka/netshoot --target=pay
You will see:
Defaulting debug container name to debugger-k2x9.
pay-7c9-x2 ~ #
What it means: Same technique on a pod.
6. Common mistakes and how Vikram avoids them
Mistake one: adding a shell and curl to the production image just to debug. Vikram keeps images small and brings tools from outside.
Mistake two: running netshoot on the default network instead of the container's network. He always uses --network container:NAME so he sees what the app sees.
Mistake three: leaving packet captures on servers. He deletes them after reading, because they may contain private data.
Mistake four: changing things before recording state. He saves logs and inspect output first.
Check yourself
Q1. Why does docker exec sh fail on distroless?
There is no shell or tools inside the image.
Q2. How does netshoot see the broken container's network?
--network container:NAME shares its network namespace.
Q3. What does nsenter -t PID -n do?
Runs a host command inside the container's network namespace.
Q4. Kubernetes equivalent of these tricks?
kubectl debug with an ephemeral container and --target.
Friends, keep production images tool-free, and bring the tools when you need them: docker debug for files and processes, netshoot for networks, nsenter when nothing else is allowed. Did you understand? Let us try it once more. Run a distroless app, break its DNS with a wrong --add-host entry, and find the problem with netshoot without rebuilding anything.
मित्रांनो, production images tool-free ठेवा, आणि गरज असेल तेव्हा tools घेऊन या: files आणि processes साठी docker debug, network साठी netshoot, काहीच परवानगी नसेल तेव्हा nsenter. एक distroless app चालवा, चुकीच्या --add-host entry ने त्याचा DNS बिघडवा, आणि काहीही rebuild न करता netshoot ने problem शोधा.
दोस्तों, production images tool-free रखिए, और ज़रूरत पर tools साथ लाइए: files और processes के लिए docker debug, network के लिए netshoot, कुछ भी allowed न हो तो nsenter. एक distroless app चलाइए, गलत --add-host entry से उसका DNS बिगाड़िए, और कुछ भी rebuild किए बिना netshoot से problem ढूँढिए.
Ravindra Bagale's Tip
Add one line to your team wiki: docker run --rm -it --network container:NAME nicolaka/netshoot. It solves half of all network tickets.
तुमच्या team wiki मध्ये एक ओळ जोडा: docker run --rm -it --network container:NAME nicolaka/netshoot. ही अर्ध्या network tickets सोडवते.
अपनी team wiki में एक line जोड़िए: docker run --rm -it --network container:NAME nicolaka/netshoot. यह आधे network tickets हल कर देती है.
Lab: try at home
- Run a distroless container and prove exec sh fails.
- Record its state and logs before debugging.
- Open docker debug if you have it, and read a config file.
- Join its network with netshoot and run dig.
- Test a port with nc and curl -v.
- Take a 50 packet tcpdump and delete it after.
- Use nsenter from the host to run ss inside the container.
Recap
- Distroless images have no shell; that is on purpose.
- docker debug brings a toolbox without changing the image.
- netshoot with --network container:NAME sees the app's network.
- nsenter and --pid container:NAME use namespaces directly.
- Debug the running container; do not rebuild to debug.