Chapter 56: Docker CI/CD with Jenkins and ECR, Deploy to Amazon ECS, Troubleshooting, Best Practices and Interview Questions

56.9 Docker disk cleanup: system df, build cache, logs and data-root

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose a Jenkins agent at Flipkart fails every build with: no space left on device. The disk is 100 GB. The team adds 50 GB more; three weeks later it is full again. The real cause is that Docker never deletes anything by itself. Every build leaves dangling images and build cache. Every stopped container keeps its writable layer. Every container with the default json-file driver keeps its logs forever, and one chatty service can write 40 GB of logs. Anonymous volumes from old test runs stay behind. In this lesson we find exactly where the space went with docker system df -v and du, clean each kind safely with filters, cap logs, schedule cleanup, and if needed move Docker's data-root to a bigger disk. The rule: measure, clean the safe things first, never delete volumes blindly.

What: every term in plain English

• docker system df — Space used by images, containers, volumes and build cache.
• Reclaimable — Space that can be freed without touching anything in use.
• Dangling image — Untagged image, usually left by rebuilding the same tag.
• Build cache — BuildKit layers and cache mounts; often the biggest item on CI agents.
• prune — Deletes unused objects of a type; --filter until=24h limits by age.
• json-file logs — Default log driver files in /var/lib/docker/containers/ID/.
• max-size / max-file — Log rotation options.
• data-root — Where Docker stores everything, default /var/lib/docker.
• Anonymous volume — Volume without a name, created by VOLUME or -v /path.

How: step by step

1. Check the disk. df -h /var/lib/docker.
2. Ask Docker. docker system df -v.
3. Find big logs. du on container log files.
4. Prune safely. Containers, dangling images, old build cache.
5. Cap logs. daemon.json log-opts.
6. Schedule cleanup. systemd timer or cron.
7. Move data-root if needed. Stop, rsync, configure, start.

Example 1: Find where the space went

Goal. Find where the space went.

Commands

df -h /var/lib/docker
docker system df
docker system df -v | sed -n '1,12p'
sudo du -sh /var/lib/docker/* 2>/dev/null | sort -h | tail -5
sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log' | sort -h | tail -3
docker ps -a --filter status=exited --format '{{.Names}} {{.Status}}' | wc -l

What each line does

• df -h — The real disk usage.
• system df — Totals and RECLAIMABLE per type.
• system df -v — Per image, container and volume details.
• du on /var/lib/docker — overlay2 (layers), containers (logs), volumes, buildkit.
• *-json.log — Container log files, often huge.
• exited count — Stopped containers still hold layers.

Green output

/dev/nvme0n1p1  100G   96G  4.0G  97% /
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          212       9         61.4GB    55.2GB (89%)
Containers      148       6         3.1GB     3.0GB (96%)
Local Volumes   37        4         8.7GB     7.9GB (90%)
Build Cache     1893      0         19.6GB    19.6GB
21G   /var/lib/docker/buildkit
42G   /var/lib/docker/overlay2
2.1G  /var/lib/docker/containers/9f1e.../9f1e...-json.log
142

Expected result. Most space is old images and build cache, plus one 2 GB log. Only 9 images and 6 containers are actually in use.

Example 2: Clean safely, in order, with filters

Goal. Clean safely, in order, with filters.

Commands

docker container prune -f --filter until=24h
docker image prune -f
docker image prune -a -f --filter until=168h --filter label!=keep=true
docker builder prune -f --filter until=72h
docker builder prune -f --keep-storage 10GB
docker volume ls -f dangling=true --format '{{.Name}}' | head
docker volume prune -f --filter label=ci-temp=true
sudo truncate -s 0 /var/lib/docker/containers/9f1e*/9f1e*-json.log
docker system df

What each line does

• container prune until=24h — Stopped for over a day.
• image prune — Dangling images only: always safe.
• prune -a until=168h — Unused images older than a week, except label keep=true.
• builder prune --keep-storage — Keeps the most useful 10 GB of cache.
• volume ls dangling — Look before deleting data.
• volume prune with label — Only volumes we marked as temporary.
• truncate log — Frees a live log without restarting; rotate properly next.

Green output

Total reclaimed space: 3.0GB
Total reclaimed space: 1.2GB
Total reclaimed space: 52.9GB
Total: 9.6GB
Total: 9.4GB
Total reclaimed space: 7.1GB
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          11        9         6.2GB     0.9GB
Build Cache     214       0         10.0GB    10.0GB

Expected result. About 85 GB freed without touching any running service or real data volume.

Example 3: Stop it coming back: log limits, a timer and a bigger data-root

Goal. Stop it coming back: log limits, a timer and a bigger data-root.

/etc/docker/daemon.json

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "20m", "max-file": "3" },
  "data-root": "/data/docker",
  "builder": { "gc": { "enabled": true, "defaultKeepStorage": "20GB" } }
}

/etc/systemd/system/docker-prune.service

[Unit]
Description=Docker cleanup
[Service]
Type=oneshot
ExecStart=/usr/bin/docker system prune -af --filter until=168h
ExecStart=/usr/bin/docker builder prune -f --keep-storage 20GB

/etc/systemd/system/docker-prune.timer

[Unit]
Description=Weekly Docker cleanup
[Timer]
OnCalendar=Sun 03:00
Persistent=true
[Install]
WantedBy=timers.target

Commands

sudo systemctl stop docker docker.socket
sudo rsync -aHAX /var/lib/docker/ /data/docker/
sudo systemctl start docker
docker info -f '{{.DockerRootDir}}'
sudo systemctl daemon-reload && sudo systemctl enable --now docker-prune.timer
systemctl list-timers docker-prune.timer --no-pager

What each line does

• max-size 20m, max-file 3 — Each container keeps at most 60 MB of logs (for new containers).
• data-root /data/docker — Docker storage on a larger disk.
• builder gc — BuildKit garbage-collects cache above 20 GB.
• rsync -aHAX while stopped — Copies layers with hard links and attributes.
• DockerRootDir — Confirms the new location.
• timer — Weekly prune of anything unused for 7 days. Volumes are not touched.

Green output

/data/docker
NEXT                        LEFT     LAST  PASSED  UNIT
Sun 2026-11-01 03:00:00 IST 6 days   -     -       docker-prune.timer

Expected result. Logs are capped, cache is bounded, storage is on a bigger disk and cleanup runs weekly. The disk-full alert stops repeating.

Every command, step by step (plain docker and Compose)

1. Overview

   docker system df
   

   What it does: Totals and reclaimable.

   Images 61.4GB 55.2GB (89%)
   

2. Details

   docker system df -v
   

   What it does: Per object.

   postgres:16 431MB shared
   

3. Safe one-liner

   docker system prune -f
   

   What it does: Stopped containers, dangling images, unused networks, dangling cache.

   Total reclaimed space: 4.1GB
   

4. Aggressive

   docker system prune -af --filter until=168h
   

   What it does: Also unused images older than a week.

   Total reclaimed space: 50GB
   

5. Big logs

   docker inspect -f '{{.LogPath}}' api | xargs sudo du -h
   

   What it does: Log file size of one container.

   2.1G ...
   

6. Volumes in use?

   docker ps -a --filter volume=pgdata
   

   What it does: Who uses a volume.

   db
   

7. Root dir

   docker info -f '{{.DockerRootDir}}'
   

   What it does: Where data lives.

   /var/lib/docker
   

When it is red 1

failed to register layer: write /var/lib/docker/...: no space left on device

Cause. Disk full during pull or build.

Fix, step by step

1. docker system df, then prune images and build cache.
2. Cap logs and add a timer.
3. Move data-root to a bigger disk.

When it is red 2

df shows free space but Docker still says no space left

Cause. Inodes are exhausted (many small files).

Fix, step by step

1. Check df -i.
2. Prune images and build cache; consider XFS with more inodes.

When it is red 3

Error response from daemon: remove pgdata: volume is in use

Cause. A container, even stopped, still uses the volume.

Fix, step by step

1. Find it with docker ps -a --filter volume=pgdata.
2. Back up, then remove the container first, only if the data is not needed.

Practice tasks

docker system df

docker image prune -f

docker container prune -f --filter until=24h

docker builder prune -f --keep-storage 10GB

sudo du -h /var/lib/docker/containers/*/*-json.log | sort -h | tail -1

"log-opts": {"max-size":"20m","max-file":"3"}

sudo systemctl enable --now docker-prune.timer

sudo rsync -aHAX /var/lib/docker/ /data/docker/

docker build --label keep=true -t base:gold .

df -h /

Recap

• Docker never cleans up by itself.
• Measure with system df -v and du.
• Prune in safe order with until filters.
• Never prune volumes blindly.
• Cap json-file logs in daemon.json.
• Schedule cleanup and size the disk for the cache you keep.

FAQ

Q. Is docker system prune safe in production?

Without -a and --volumes it is mostly safe; still run it with filters and on a schedule you control.

Q. Do log options apply to running containers?

Only to containers created after the change.

Q. What about Docker Desktop?

Use the Resources settings to limit the disk image size, and prune the same way.

Q. Should I use the local log driver?

Yes, it rotates and compresses by default; but docker logs still works.

Q. How do I clean Kubernetes nodes?

kubelet garbage-collects images by disk thresholds; use crictl rmi --prune manually if needed.

Q. Can I prune by label?

Yes: --filter label=ci-temp=true, or label!=keep=true.

Q. How much disk should a CI agent have?

Enough for base images plus the build cache you keep, usually 100 to 200 GB, with builder gc and a weekly prune so it never grows without limit.

Interview questions

1. A server ran out of disk because of Docker. What do you check?

   Model answer: df, docker system df -v, du on overlay2, buildkit and container logs; then prune safely and cap logs.

2. What is reclaimable space?

   Model answer: Space held by objects not used by any container.

3. How do you stop log files filling the disk?

   Model answer: max-size and max-file, the local driver, or ship logs elsewhere.

4. How do you move Docker's storage?

   Model answer: Stop Docker, rsync to the new path, set data-root, start, verify.

5. Difference between image prune and image prune -a?

   Model answer: Without -a only dangling; with -a all images not used by a container.

6. How do you keep build cache bounded?

   Model answer: builder prune --keep-storage or BuildKit gc settings.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview card

मराठी cards