Chapter 56: Docker CI/CD with Jenkins and ECR, Deploy to Amazon ECS, Troubleshooting, Best Practices and Interview Questions

56.7 Debug slim and distroless containers: docker debug and netshoot

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we are at PhonePe. Our payment service runs on a distroless image, as chapter 55 taught us. Now it cannot reach the database, and we try docker exec -it pay sh. The answer: executable file not found. There is no shell, no curl, no ping, no nslookup. That is good for security and bad for debugging at 2 am. We must not add debug tools to the production image; that would undo the hardening. Instead we bring the tools to the container from outside. Two ways: docker debug, which attaches a toolbox shell to any container or image without changing it, and nicolaka/netshoot, a network toolbox image that we start inside the same network or PID namespace as the broken container. The same idea exists in Kubernetes as kubectl debug with ephemeral containers. In this lesson we debug a distroless service's DNS, ports, TLS and processes without touching its image.

What: every term in plain English

• Distroless — Image with only the app and runtime; no shell or package manager.
• docker debug — Docker CLI command that opens a toolbox shell into any container or image; tools are not added to it.
• netshoot — nicolaka/netshoot: image with curl, dig, tcpdump, ss, iperf3, mtr, openssl and more.
• --network container:NAME — Join another container's network namespace: same IP, ports and DNS.
• --pid container:NAME — See another container's processes.
• Sidecar — A helper container placed next to the main one.
• tcpdump — Captures packets on an interface.
• ss — Shows listening sockets and connections.
• kubectl debug — Kubernetes equivalent using ephemeral containers.

How: step by step

1. Confirm there is no shell. docker exec fails.
2. Try docker debug. Toolbox shell inside the container's view.
3. Start netshoot in its network. --network container:pay.
4. Check DNS, ports, TLS. dig, ss, curl, openssl.
5. Check processes. --pid container:pay.
6. Capture traffic if needed. tcpdump to a file.
7. Remove the toolbox. --rm; the target image is unchanged.

Example 1: A distroless service that we cannot exec into

Goal. A distroless service that we cannot exec into.

compose.yaml

services:
  pay:
    image: gcr.io/distroless/python3-debian12:nonroot
    command: ["-m", "http.server", "8080"]
    environment:
      DB_HOST: postgres-db
    networks: [backend]
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: change-me
    networks: [backend]
networks:
  backend: {}

Commands

docker compose up -d
docker compose exec pay sh
docker compose exec pay ls
docker inspect -f '{{.Config.User}} {{json .Config.Entrypoint}}' $(docker compose ps -q pay)

What each line does

• distroless python3 nonroot — Python runtime only, runs as UID 65532.
• DB_HOST: postgres-db — A deliberate bug: the service is called db.
• exec sh — Fails: no shell exists.
• exec ls — Fails too: no ls.
• inspect — We can still read config from outside.

Green output

OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH
OCI runtime exec failed: exec failed: unable to start container process: exec: "ls": executable file not found in $PATH
nonroot ["/usr/bin/python3.11"]

Expected result. This is the normal situation with hardened images. We need tools from outside.

Example 2: netshoot in the same network namespace finds the DNS bug

Goal. netshoot in the same network namespace finds the DNS bug.

Commands

PAY=$(docker compose ps -q pay)
docker run --rm -it --network container:$PAY nicolaka/netshoot:v0.13
# inside netshoot:
ss -ltnp
dig +short postgres-db
dig +short db
nc -zv db 5432
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080
exit
docker run --rm --network container:$PAY nicolaka/netshoot:v0.13 \
  tcpdump -i eth0 -c 20 -w /dev/stdout port 5432 > db.pcap

What each line does

• --network container:$PAY — netshoot shares pay's IP, ports and DNS resolver.
• ss -ltnp — pay listens on 8080, so the app is up.
• dig postgres-db — Empty: no such name. That is the bug.
• dig db — Resolves: the right name is db.
• nc -zv db 5432 — Port reachable.
• tcpdump to db.pcap — Capture for Wireshark if deeper analysis is needed.

Green output

State  Recv-Q Send-Q Local Address:Port  Peer Address:Port
LISTEN 0      5      0.0.0.0:8080        0.0.0.0:*
(no output for postgres-db)
172.22.0.2
Connection to db (172.22.0.2) 5432 port [tcp/postgresql] succeeded!
200

Expected result. Two minutes of netshoot found it: DB_HOST points to a name that does not exist. Fix DB_HOST: db and redeploy. The pay image never changed.

Example 3: docker debug and a PID-namespace toolbox for processes and files

Goal. docker debug and a PID-namespace toolbox for processes and files.

Commands

docker debug $(docker compose ps -q pay)
# inside the debug shell (tools come from the toolbox, files are the container's):
ps aux
cat /proc/1/environ | tr '\0' '\n' | grep DB_
ls -l /usr/lib/python3.11 | head -3
exit

docker run --rm -it --pid container:$(docker compose ps -q pay) --network container:$(docker compose ps -q pay) \
  nicolaka/netshoot:v0.13 sh -c 'ps aux; ls /proc/1/root/ | head; cat /proc/1/root/etc/os-release | head -2'

docker debug gcr.io/distroless/python3-debian12:nonroot

What each line does

• docker debug CONTAINER — Opens a shell with tools, seeing the container's filesystem and processes.
• /proc/1/environ — Real environment of PID 1: confirms DB_HOST.
• --pid container: — netshoot sees pay's processes.
• /proc/1/root — The target container's root filesystem, from the toolbox.
• docker debug IMAGE — Explore an image that is not running.
• Exit — Toolbox disappears; the container is unchanged.

Green output

PID   USER     COMMAND
1     nonroot  /usr/bin/python3.11 -m http.server 8080
DB_HOST=postgres-db
PID   USER     TIME  COMMAND
    1 65532     0:00 /usr/bin/python3.11 -m http.server 8080
bin  boot  dev  etc  home  lib ...
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"

Expected result. Both tools let us read env, processes and files of a shell-less container. docker debug is easiest when available; netshoot with --pid and --network works on any Docker host.

Every command, step by step (plain docker and Compose)

1. Ports listening

   docker run --rm --network container:pay nicolaka/netshoot:v0.13 ss -ltn
   

   What it does: Sockets in pay's namespace.

   LISTEN 0.0.0.0:8080
   

2. DNS

   docker run --rm --network container:pay nicolaka/netshoot:v0.13 dig +short db
   

   What it does: Name resolution as pay sees it.

   172.22.0.2
   

3. HTTP

   docker run --rm --network container:pay nicolaka/netshoot:v0.13 curl -sv http://db:5432
   

   What it does: Raw HTTP test.

   Connected to db
   

4. TLS

   docker run --rm nicolaka/netshoot:v0.13 openssl s_client -connect api.razorpay.com:443 -brief
   

   What it does: Certificate check.

   Protocol version: TLSv1.3
   

5. Path

   docker run --rm nicolaka/netshoot:v0.13 mtr -rwc 5 8.8.8.8
   

   What it does: Route and loss.

   HOST: ... Loss% 0.0
   

6. Host network

   docker run --rm --network host nicolaka/netshoot:v0.13 ss -ltn
   

   What it does: Host sockets.

   LISTEN 0.0.0.0:22
   

7. Kubernetes

   kubectl debug -it pay-7d9f --image=nicolaka/netshoot --target=pay
   

   What it does: Ephemeral debug container.

   Defaulting debug container name to debugger-x2k
   

When it is red 1

docker: 'debug' is not a docker command.

Cause. docker debug is a Docker Desktop and paid-plan feature.

Fix, step by step

1. Update Docker Desktop and sign in, or use netshoot with --pid and --network container:.

When it is red 2

docker: Error response from daemon: cannot join network of a non running container

Cause. The target container has stopped.

Fix, step by step

1. docker logs it first.
2. Run the image with --entrypoint override using a debug tag, or docker debug IMAGE.

When it is red 3

tcpdump: eth0: You don't have permission to capture on that device

Cause. Missing NET_RAW / NET_ADMIN capability.

Fix, step by step

1. Add --cap-add NET_ADMIN --cap-add NET_RAW to the netshoot container only.

Practice tasks

docker compose exec pay sh

docker run --rm -it --network container:PAY nicolaka/netshoot:v0.13

ss -ltnp

dig +short postgres-db

nc -zv db 5432

--pid container:PAY ... ps aux

tr '\0' '\n' with /proc/1/environ

tcpdump -i eth0 -c 20 -w /dev/stdout

docker debug gcr.io/distroless/python3-debian12:nonroot

docker compose up -d

Recap

• Hardened images have no shell, by design.
• Bring tools from outside: docker debug or netshoot.
• --network container: shares IP, ports and DNS.
• --pid container: shares processes; /proc/1/root shows files.
• Give NET_ADMIN only to the toolbox.
• In Kubernetes use kubectl debug.

FAQ

Q. Is docker debug free?

It ships with Docker Desktop; check the current plan terms in Docker's docs.

Q. Is netshoot safe to run in production?

Run it briefly, with --rm, and only on hosts you own; it has powerful tools.

Q. Can I debug a crashed container?

Not its namespaces; run the image again with docker debug IMAGE or a debug tag.

Q. What is busybox:debug?

Distroless ships :debug tags with a busybox shell for non-production use.

Q. Does this work in ECS?

Use ECS Exec, or add a netshoot sidecar to the task definition temporarily.

Q. Can I copy files out of distroless?

Yes, docker cp works without a shell.

Q. Which tools inside netshoot matter most?

dig and nslookup for DNS, ss for ports, curl for HTTP, nc for raw TCP, openssl for TLS, tcpdump for packets, iperf3 for bandwidth.

Interview questions

1. How do you debug a container with no shell?

   Model answer: docker debug, or a netshoot container sharing its network and PID namespaces, or kubectl debug in Kubernetes.

2. What does --network container:x do?

   Model answer: Puts the new container in x's network namespace: same interfaces, IP, ports and resolver.

3. Why keep debug tools out of images?

   Model answer: Smaller attack surface and fewer CVEs.

4. How would you capture traffic of one container?

   Model answer: tcpdump from netshoot in its network namespace with NET_RAW.

5. What are ephemeral containers?

   Model answer: Kubernetes debug containers added to a running pod without restart.

6. How do you read a distroless container's env?

   Model answer: docker inspect, or /proc/1/environ from a PID-sharing toolbox.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview card

मराठी cards