56.10 Disk cleanup and log limits: stop Docker from filling the disk
Friends, the number one reason a Docker server suddenly stops working is not a bug in the app. It is a full disk. Old images, stopped containers, build cache, forgotten volumes and huge log files pile up quietly until the disk hits 100 percent and everything fails at once. Today, step by step: measure what uses space, clean each type safely, set log limits so it never happens again, and automate the cleanup.
मित्रांनो, Docker server अचानक बंद पडण्याचं पहिलं कारण app मधला bug नसतो. ते असतं भरलेली disk. जुन्या images, थांबलेले containers, build cache, विसरलेले volumes आणि मोठ्या log files शांतपणे साठत जातात, disk 100 टक्के होते आणि सगळं एकदम fail होतं. आज step by step: जागा कशाने वापरली ते मोजा, प्रत्येक प्रकार सुरक्षितपणे साफ करा, log limits सेट करा म्हणजे हे पुन्हा होणार नाही, आणि cleanup automate करा.
दोस्तों, Docker server अचानक बंद होने का पहला कारण app का bug नहीं होता. वह होती है भरी हुई disk. पुरानी images, रुके हुए containers, build cache, भूले हुए volumes और बड़ी log files चुपचाप जमा होते हैं, disk 100 प्रतिशत हो जाती है और सब एक साथ fail हो जाता है. आज step by step: जगह किसने ली यह नापिए, हर प्रकार को सुरक्षित तरीके से साफ़ कीजिए, log limits सेट कीजिए ताकि ऐसा दोबारा न हो, और cleanup automate कीजिए.
1. Start from a story
At Paytm, a Jenkins build agent stopped every few weeks with no space left on device. Someone would log in, run docker system prune -a --volumes, and the problem went away for a while.
One day that same command deleted a volume holding a test database that another team needed for a week of performance tests. Nobody had told them that prune with --volumes deletes data.
A senior engineer named Meera wrote a proper plan. Measure first with docker system df. Clean containers, images and build cache with filters by age. Never prune volumes automatically. Add log limits in daemon.json. Run the safe cleanup from a nightly timer. The agent never filled up again and no data was lost.
Today we follow Meera's plan.
2. Measure first
Step 1: df and docker system df
Why: How full, and who is using it.
What: Start with the host disk, then ask Docker which type is using space. The RECLAIMABLE column shows what cleanup could free.
Example 1: Host disk
Type this:
df -h /var/lib/docker | tail -1
You will see:
/dev/xvdb 50G 47G 3.0G 95% /var/lib/docker
What it means: 95 percent full. Act now.
Example 2: Docker summary
Type this:
docker system df
You will see:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 84 9 31.2GB 26.8GB (85%)
Containers 41 6 2.1GB 1.9GB (90%)
Local Volumes 12 4 6.4GB 2.2GB (34%)
Build Cache 310 0 5.9GB 5.9GB
What it means: Images and build cache are the big ones.
Example 3: Details
Type this:
docker system df -v | sed -n '/Images space usage/,/^$/p' | head -5
You will see:
REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERS
shop-api old1 ... 5 weeks ago 1.1GB ... 0
What it means: Per image, with how many containers use it.
Step 2: Find big log files
Why: Logs hide in another folder.
What: With the default json-file driver and no limit, each container's log grows forever under /var/lib/docker/containers. system df does not show it.
Example 4: Biggest logs
Type this:
sudo du -h /var/lib/docker/containers/*/*-json.log | sort -h | tail -3
You will see:
1.2G .../8a1c...-json.log
3.4G .../f02b...-json.log
9.8G .../c71d...-json.log
What it means: One container wrote almost 10 GB.
Example 5: Which container
Type this:
docker ps -a --no-trunc --format '{{.ID}} {{.Names}}' | grep c71d
You will see:
c71d... worker
What it means: The chatty worker.
Example 6: Its log setting
Type this:
docker inspect --format '{{.HostConfig.LogConfig}}' worker
You will see:
{json-file map[]}
What it means: No max-size. That is the bug.
3. Clean each type safely
Step 3: Stopped containers
Why: Safe to remove.
What: Stopped containers keep their writable layer and logs. container prune removes only stopped ones. Add an age filter so you keep recent ones for debugging.
Example 7: See them
Type this:
docker ps -a --filter status=exited --format '{{.Names}} {{.Status}}' | head -3
You will see:
build-41 Exited (0) 3 weeks ago
build-42 Exited (0) 3 weeks ago
test-db Exited (0) 2 hours ago
What it means: Old build containers.
Example 8: Prune older than a day
Type this:
docker container prune -f --filter until=24h
You will see:
Total reclaimed space: 1.8GB
What it means: Recent ones kept.
Example 9: Check
Type this:
docker ps -a --filter status=exited -q | wc -l
You will see:
1
What it means: Only the recent one left.
Step 4: Dangling and unused images
Why: The largest win.
What: Dangling images have no tag, left after rebuilds. image prune removes them. With -a it removes every image not used by a container, so add an age filter.
Example 10: Dangling
Type this:
docker image prune -f
You will see:
Deleted Images:
deleted: sha256:5e2a...
Total reclaimed space: 4.1GB
What it means: Untagged layers gone.
Example 11: Unused older than a week
Type this:
docker image prune -a -f --filter until=168h
You will see:
Total reclaimed space: 21.3GB
What it means: Images in use stay.
Example 12: Keep important ones
Type this:
docker image prune -a -f --filter until=168h --filter 'label!=keep=true'
You will see:
Total reclaimed space: 0B
What it means: Images labelled keep=true are never removed.
Step 5: Build cache
Why: Grows with every build.
What: BuildKit stores cache for faster builds. builder prune removes it. Keep recent cache so builds stay fast, or cap it by size.
Example 13: See it
Type this:
docker buildx du | tail -2
You will see:
Shared: 1.1GB
Total: 5.9GB
What it means: Almost 6 GB of cache.
Example 14: Prune older than 3 days
Type this:
docker builder prune -f --filter until=72h
You will see:
Total: 4.6GB
What it means: Recent cache kept for speed.
Example 15: Cap by size
Type this:
docker builder prune -f --keep-storage 2GB
You will see:
Total: 0.4GB
What it means: Leaves at most 2 GB.
Step 6: Unused networks
Why: Small but tidy.
What: Compose projects leave networks behind. network prune removes those with no containers.
Example 16: List
Type this:
docker network ls --format '{{.Name}}' | wc -l
You will see:
27
What it means: Many old networks.
Example 17: Prune
Type this:
docker network prune -f --filter until=24h
You will see:
Deleted Networks:
build41_default
build42_default
What it means: Only unused ones.
Example 18: Check
Type this:
docker network ls --format '{{.Name}}' | wc -l
You will see:
6
What it means: Back to normal.
Step 7: Volumes: the careful one
Why: Volumes hold data.
What: Never prune volumes automatically. List dangling ones, check what is inside, back up if unsure, and remove by name.
Example 19: Dangling volumes
Type this:
docker volume ls -f dangling=true --format '{{.Name}}'
You will see:
perf-testdb
3f9e2c1a...
What it means: Not attached to any container, but may still matter.
Example 20: Look inside first
Type this:
docker run --rm -v perf-testdb:/v alpine sh -c 'du -sh /v; ls /v | head -3'
You will see:
2.1G /v
base
global
pg_wal
What it means: A database. Ask the owner before deleting.
Example 21: Back up, then remove by name
Type this:
docker run --rm -v 3f9e2c1a:/v -v "$PWD":/b alpine tar czf /b/vol-3f9e.tgz -C /v . && docker volume rm 3f9e2c1a
You will see:
3f9e2c1a
What it means: Removed only the one we checked.
4. Log limits
Step 8: Limit one container
Why: Fix the chatty worker.
What: --log-opt max-size and max-file rotate the json-file log. Log options apply when a container is created, so recreate it.
Example 22: Recreate with limits
Type this:
docker rm -f worker && docker run -d --name worker --log-opt max-size=10m --log-opt max-file=3 shop-worker:2.1
You will see:
worker
9d0e...
What it means: At most 30 MB of logs.
Example 23: Check
Type this:
docker inspect --format '{{.HostConfig.LogConfig.Config}}' worker
You will see:
map[max-file:3 max-size:10m]
What it means: Limits in place.
Example 24: Compose version
Type this:
services:
worker:
image: shop-worker:2.1
logging:
driver: json-file
options: { max-size: "10m", max-file: "3" }
You will see:
(compose.yaml)
What it means: Same limits in the file.
Step 9: Default limits for every container
Why: Set once in daemon.json.
What: Put log-driver and log-opts in /etc/docker/daemon.json and restart Docker. New containers get the limits. The local driver also compresses logs.
Example 25: Write daemon.json
Type this:
printf '%s\n' '{' ' "log-driver": "local",' ' "log-opts": { "max-size": "10m", "max-file": "3" }' '}' | sudo tee /etc/docker/daemon.json
You will see:
{
"log-driver": "local",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
What it means: Valid JSON, written without a heredoc.
Example 26: Restart and check
Type this:
sudo systemctl restart docker && docker info --format '{{.LoggingDriver}}'
You will see:
local
What it means: The new default.
Example 27: Old containers keep old settings
Type this:
docker inspect --format '{{.Name}} {{.HostConfig.LogConfig.Type}}' $(docker ps -q) | head -2
You will see:
/api json-file
/worker json-file
What it means: Recreate them to pick up the default.
Step 10: Emergency: truncate a huge log
Why: When the disk is at 100 percent now.
What: If you cannot recreate the container right now, truncate its log file to zero. Do not delete the file, because Docker keeps it open. Then add limits properly.
Example 28: Find the path
Type this:
docker inspect --format '{{.LogPath}}' worker
You will see:
/var/lib/docker/containers/c71d.../c71d...-json.log
What it means: The exact file.
Example 29: Truncate
Type this:
sudo truncate -s 0 $(docker inspect --format '{{.LogPath}}' worker) && df -h /var/lib/docker | tail -1
You will see:
/dev/xvdb 50G 14G 36G 28% /var/lib/docker
What it means: Space back immediately.
Example 30: Logs still work
Type this:
docker logs --tail 1 worker
You will see:
job 88231 done
What it means: Docker keeps writing to the same file.
5. Automate it
Step 11: A nightly cleanup timer
Why: So nobody has to remember.
What: Put the safe prunes in a script and run it every night with a systemd timer or cron. No volume prune in the script.
Example 31: The script
Type this:
printf '%s\n' '#!/bin/sh' 'docker container prune -f --filter until=24h' 'docker image prune -a -f --filter until=168h --filter label!=keep=true' 'docker builder prune -f --keep-storage 5GB' 'docker network prune -f --filter until=24h' | sudo tee /usr/local/bin/docker-clean >/dev/null && sudo chmod 0755 /usr/local/bin/docker-clean && cat /usr/local/bin/docker-clean | wc -l
You will see:
5
What it means: Four safe prunes, mode 0755.
Example 32: Cron entry
Type this:
printf '%s\n' '30 2 * * * root /usr/local/bin/docker-clean >> /var/log/docker-clean.log 2>&1' | sudo tee /etc/cron.d/docker-clean && sudo chmod 0644 /etc/cron.d/docker-clean
You will see:
30 2 * * * root /usr/local/bin/docker-clean >> /var/log/docker-clean.log 2>&1
What it means: Runs at 2:30 every night.
Example 33: Disk alert
Type this:
[ $(df --output=pcent /var/lib/docker | tail -1 | tr -dc 0-9) -gt 80 ] && echo 'ALERT: docker disk over 80%' || echo ok
You will see:
ok
What it means: Add this to monitoring.
Step 12: Move Docker to a bigger disk
Why: When cleanup is not enough.
What: If the server really needs more space, attach a bigger disk and move the Docker data root there with the data-root setting in daemon.json. Stop Docker first, copy the folder with rsync so nothing is lost, then start it again and check.
Example 34: Stop and copy
Type this:
sudo systemctl stop docker docker.socket && sudo rsync -aHAX /var/lib/docker/ /data/docker/ && sudo du -sh /data/docker
You will see:
19G /data/docker
What it means: All images, containers and volumes copied.
Example 35: Point Docker at it
Type this:
printf '%s\n' '{' ' "data-root": "/data/docker",' ' "log-driver": "local",' ' "log-opts": { "max-size": "10m", "max-file": "3" }' '}' | sudo tee /etc/docker/daemon.json >/dev/null && sudo systemctl start docker && docker info --format '{{.DockerRootDir}}'
You will see:
/data/docker
What it means: Docker now lives on the big disk.
Example 36: Check everything is there
Type this:
docker image ls -q | wc -l && docker volume ls -q | wc -l
You will see:
9
12
What it means: Same images and volumes. Remove the old folder only after a day of normal running.
6. Common mistakes and how Meera avoids them
Mistake one: docker system prune -a --volumes on a shared server. Meera never prunes volumes automatically.
Mistake two: deleting a log file with rm while the container runs. The space is not freed until the container stops. She uses truncate.
Mistake three: setting log limits in daemon.json and expecting old containers to change. She recreates them.
Mistake four: cleaning only when the disk is full. She runs the timer every night and alerts at 80 percent.
Check yourself
Q1. Which command shows Docker disk use by type?
docker system df.
Q2. Why never prune volumes automatically?
Volumes hold real data that may still be needed.
Q3. How do you limit logs for every new container?
log-driver and log-opts max-size and max-file in daemon.json.
Q4. How do you free a huge log right now?
truncate -s 0 on the LogPath, not rm.
Friends, measure with df and docker system df, prune containers, images, cache and networks with age filters, handle volumes by hand, set log limits in daemon.json and automate it all every night. Did you understand? Let us try it once more. On a lab machine, fill the disk with builds and a noisy container, then bring it back under 30 percent using only the safe commands.
मित्रांनो, df आणि docker system df ने मोजा, age filters सह containers, images, cache आणि networks prune करा, volumes हाताने हाताळा, daemon.json मध्ये log limits सेट करा आणि हे सगळं रोज रात्री automate करा. lab machine वर builds आणि गोंगाट करणाऱ्या container ने disk भरा, मग फक्त सुरक्षित commands वापरून ती 30 टक्क्यांखाली आणा.
दोस्तों, df और docker system df से नापिए, age filters के साथ containers, images, cache और networks prune कीजिए, volumes हाथ से संभालिए, daemon.json में log limits सेट कीजिए और यह सब हर रात automate कीजिए. lab machine पर builds और शोर करने वाले container से disk भरिए, फिर सिर्फ सुरक्षित commands से उसे 30 प्रतिशत से नीचे लाइए.
Ravindra Bagale's Tip
Put the log limits in daemon.json on the day you install Docker. It takes one minute and prevents the most common outage.
Docker install केल्याच्या दिवशीच daemon.json मध्ये log limits ठेवा. याला एक मिनिट लागतो आणि सर्वात सामान्य outage टळतो.
Docker install करने के दिन ही daemon.json में log limits रखिए. इसमें एक मिनट लगता है और सबसे आम outage टल जाता है.
Lab: try at home
- Run df and docker system df and write down each type.
- Find the largest container log file.
- Prune containers, images and build cache with age filters.
- Inspect one dangling volume before deciding.
- Recreate one container with log limits.
- Set default log limits in daemon.json.
- Install the nightly cleanup script and an 80 percent alert.
Recap
- Measure first: df, docker system df, du on container logs.
- Prune with until filters; keep recent things.
- Volumes are data: check, back up, remove by name.
- Log limits in daemon.json; recreate old containers.
- Automate safe cleanup and alert at 80 percent.