Guides

Fix "No Space Left on Device" on Linux, Docker and EC2

By Ravindra Bagale, AWS & DevOps trainer · LinkedIn · Published: 16 October 2026 · Last updated: 17 October 2026

"No space left on device" means a Linux filesystem has run out of free blocks or free inodes, so no new file can be written. Run df -h to find which filesystem is at 100%, and df -i to check inodes. Then use du to find the biggest folder. On Docker hosts it is usually images, build cache or container logs. Clean those safely, add log rotation, or grow the EBS volume.

df -h                                        # which filesystem is 100%?
df -i                                        # out of inodes?
sudo du -xh --max-depth=1 / | sort -h | tail # biggest top folders
docker system df                             # Docker images, containers, cache
sudo journalctl --vacuum-size=200M           # shrink system logs

A live example first

A Jenkins build agent on EC2 (8 GB root disk) fails in the middle of a build:

failed to register layer: write /usr/lib/jvm/...: no space left on device

Step one, measure:

df -h /

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  8.0G  8.0G   20K 100% /

Step two, find the folder:

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -5

310M    /usr
520M    /home
1.1G    /var/log
5.6G    /var/lib
8.0G    /

docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          41        3         4.2GB     3.9GB (92%)
Containers      3         3         12MB      0B
Local Volumes   2         1         180MB     60MB
Build Cache     118       0         1.1GB     1.1GB

Old images and build cache use 5 GB. Clean them:

docker image prune -a -f
docker builder prune -f
df -h /

/dev/nvme0n1p1  8.0G  3.1G  4.9G  39% /

The build runs again. Friends, notice we did not delete anything at random. We measured, then removed only what Docker said was reclaimable.

Why does Linux run out of space?

A filesystem has two limits. Blocks hold the data, and df -h shows them. Inodes are one per file or folder, and df -i shows them. You can run out of either one. A cache folder with 2 million tiny session files can use 100% of inodes while df -h still shows free gigabytes.

How do I fix it step by step?

1. Run df -h and find the filesystem at 100%.
2. Run df -i on the same filesystem.
3. Run sudo du -xh --max-depth=1 <mount> | sort -h | tail and go one folder deeper each time.
4. If /var/lib/docker is big, use the Docker steps below.
5. If /var/log is big, use the log steps below.
6. If nothing adds up, run sudo lsof +L1 to find deleted files still held open.
7. If the data is all needed, grow the disk.

Clean Docker safely

Never run docker system prune -a --volumes on a server with database containers unless you have checked every volume.

Stop Docker logs from filling the disk

The default json-file log driver has no size limit. One chatty container can write gigabytes. Set limits in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

Restart Docker. The limit applies to new containers, so recreate the old ones.

Clean system and app logs

sudo journalctl --disk-usage
sudo journalctl --vacuum-size=200M
sudo du -sh /var/log/* | sort -h | tail

For app logs, use logrotate. Each stack has its own usual log location:

• nginx / Apache: /var/log/nginx/, /var/log/httpd/, rotated by logrotate by default.
• Tomcat / Java: catalina.out grows forever unless rotated. Add a logrotate rule with copytruncate.
• Node.js with PM2: install pm2-logrotate.
• Python / Gunicorn: use RotatingFileHandler or log to stdout and let journald handle it.
• MySQL: old binary logs. Set binlog_expire_logs_seconds.
• PostgreSQL: WAL files pile up when a replication slot is stuck. Check pg_replication_slots.
• MongoDB: use db.adminCommand({logRotate: 1}).

The deleted-but-open file trap

If you rm a 3 GB log while nginx still writes to it, the space is not freed until nginx closes it:

sudo lsof +L1

nginx  1234  root  5w  REG  259,1  3221225472  0  /var/log/nginx/access.log (deleted)

Run sudo systemctl reload nginx, and the space comes back. Next time, empty a live log with sudo truncate -s 0 file instead of rm.

Grow the EBS volume on EC2

1. In EC2 → Volumes, select the volume → Modify → set the new size (for example 20 GiB).
2. Wait until the state is optimizing or completed.
3. Run lsblk to see the bigger disk.
4. Run sudo growpart /dev/nvme0n1 1.
5. On XFS (Amazon Linux), run sudo xfs_growfs -d /.
6. On ext4 (Ubuntu), run sudo resize2fs /dev/nvme0n1p1.
7. Run df -h to confirm.

No reboot is needed. The same idea works on Azure (expand the managed disk) and GCP (gcloud compute disks resize).

Real example

Elasticsearch protects itself from full disks with "disk watermarks". At the flood-stage watermark (95% by default), it marks indices read-only, and writes fail until space is freed. Many teams first meet this when Kibana suddenly stops saving data. It is the same "no space" problem, handled by the product before Linux itself returns error 28. Watch disk use with CloudWatch or Prometheus alerts at 80%, long before any watermark.

Summary table