Chapter 32: Jenkins Governance, Troubleshooting and Expert Projects

32.3 Disk full

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we are at Ola and the Jenkins disk fills up at 3 a.m. Builds fail with strange errors. Jenkins may even stop. Disks fill slowly: old build logs, old artifacts, workspaces of deleted branches, Docker images on agents. Nobody notices until it is 100 percent. The fix has two parts. First, free space safely now. Second, set rules so it never fills again: keep only N builds, clean workspaces, prune Docker, and alert at 80 percent.

What we will build

You will find what eats the disk, free it safely, and add lasting rules. You will do it on the controller (builds and artifacts), on agents (workspaces and Docker), and with an alert in CloudWatch and a cron job.

What: every term in plain English

• JENKINS_HOME — The folder with all Jenkins data, like /var/lib/jenkins.
• builds/ — Each job keeps one folder per build with logs and artifacts. This grows fast.
• workspace — The folder where an agent checks out code and builds. Can be large.
• buildDiscarder — A Jenkinsfile option that deletes old builds automatically.
• cleanWs() — A step from Workspace Cleanup plugin that deletes the workspace after a build.
• docker system prune — Deletes stopped containers, unused networks and dangling images.
• du — Disk usage command that shows folder sizes.

How: step by step

1. See how full the disk is

   df -h /var/lib/jenkins
   

   What it does: Start with numbers.

   Output

   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme1n1    100G   94G  6.0G  94% /var/lib/jenkins
   

2. Find the biggest folders

   sudo du -xh --max-depth=1 /var/lib/jenkins | sort -rh | head -5
   

   What it does: Shows where the space went.

   Output

   94G  /var/lib/jenkins
   71G  /var/lib/jenkins/jobs
   15G  /var/lib/jenkins/workspace
   5.2G /var/lib/jenkins/caches
   

3. Find the biggest jobs

   sudo du -sh /var/lib/jenkins/jobs/* | sort -rh | head -3
   

   What it does: Usually two or three jobs use most space.

   Output

   38G  /var/lib/jenkins/jobs/android-app
   12G  /var/lib/jenkins/jobs/data-export
   4.1G /var/lib/jenkins/jobs/web
   

4. Count builds of the worst job

   ls /var/lib/jenkins/jobs/android-app/builds | wc -l
   

   What it does: Thousands of builds means no discard rule.

   Output

   4128
   

5. Free space with the Script Console safely

   Manage Jenkins > Script Console: Jenkins.instance.getItemByFullName("android-app").builds.drop(30).each{ it.delete() }
   

   What it does: Builds are newest first, so drop(30) keeps the newest 30 and deletes the rest. Jenkins updates its memory too. Do not rm folders while Jenkins runs.

   Output

   Result: [android-app #4098, android-app #4097, ...]
   

6. Check space again

   df -h /var/lib/jenkins
   

   What it does: Confirm the result.

   Output

   /dev/nvme1n1  100G  58G  42G  58% /var/lib/jenkins
   

7. Add a discard rule so it stays fixed

   options { buildDiscarder(logRotator(numToKeepStr: "30", artifactNumToKeepStr: "5")) }
   

   What it does: Put it in the Jenkinsfile of every job, or in the shared library.

   Output

   (no output)
   

Example 1: Controller: keep 30 builds and 5 with artifacts, and clean the workspace after each build

Stack. Android Gradle, Workspace Cleanup plugin

Goal. Controller: keep 30 builds and 5 with artifacts, and clean the workspace after each build.

Jenkinsfile

pipeline {
  agent { label 'android' }
  options {
    buildDiscarder(logRotator(numToKeepStr: '30', daysToKeepStr: '30', artifactNumToKeepStr: '5'))
    disableConcurrentBuilds()
  }
  stages {
    stage('Build APK') { steps { sh './gradlew assembleRelease' } }
    stage('Keep APK') { steps { archiveArtifacts artifacts: 'app/build/outputs/apk/release/*.apk', fingerprint: true } }
  }
  post {
    always { cleanWs(deleteDirs: true, notFailBuild: true) }
  }
}

What each line does

• numToKeepStr: '30' — Keep the last 30 builds, delete older ones.
• daysToKeepStr: '30' — Also delete builds older than 30 days.
• artifactNumToKeepStr: '5' — Only the last 5 builds keep their APK files. Logs stay for 30.
• disableConcurrentBuilds() — Two builds at once would use two workspaces.
• cleanWs(deleteDirs: true) — Delete the workspace after the build.
• notFailBuild: true — A cleanup error does not turn the build red.

Green output

[Pipeline] archiveArtifacts
Archiving artifacts
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] done
Finished: SUCCESS

Expected result. The job never keeps more than 30 builds or 5 APK sets. Disk use for this job drops from 38 GB to about 2 GB.

Example 2: Agents: a nightly cron job that prunes Docker and old workspaces

Stack. Ubuntu agents, cron, Docker

Goal. Agents: a nightly cron job that prunes Docker and old workspaces.

/etc/cron.d/agent-cleanup

# m h dom mon dow user command
30 2 * * * root docker system prune -af --filter "until=72h" > /var/log/agent-cleanup.log 2>&1
45 2 * * * root find /home/jenkins/agent/workspace -maxdepth 1 -type d -mtime +3 -exec rm -rf {} + >> /var/log/agent-cleanup.log 2>&1

What each line does

• 30 2 * * * — Run every day at 2:30 a.m.
• docker system prune -af — Remove unused images, stopped containers and build cache.
• --filter "until=72h" — Keep anything used in the last 3 days, so builds stay fast.
• find ... -mtime +3 — Workspaces not changed for 3 days.
• -exec rm -rf {} + — Delete them. Agents only, never on JENKINS_HOME.

Green output

$ cat /var/log/agent-cleanup.log
Deleted Images:
untagged: payments-api:7c1e9a2
deleted: sha256:4b9d...
Total reclaimed space: 23.4GB

Expected result. Agent disk drops from 91 to 47 percent overnight, every night.

Example 3: Alert before it is full: CloudWatch agent disk metric with an alarm at 80 percent

Stack. AWS EC2, CloudWatch agent, SNS

Goal. Alert before it is full: CloudWatch agent disk metric with an alarm at 80 percent.

terminal

aws cloudwatch put-metric-alarm \
  --alarm-name jenkins-disk-80 \
  --namespace CWAgent --metric-name disk_used_percent \
  --dimensions Name=InstanceId,Value=i-0abc1234def567890 Name=path,Value=/var/lib/jenkins Name=device,Value=nvme1n1 Name=fstype,Value=xfs \
  --statistic Average --period 300 --evaluation-periods 2 \
  --threshold 80 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:ap-south-1:111122223333:ci-oncall
aws cloudwatch describe-alarms --alarm-names jenkins-disk-80 --query "MetricAlarms[0].StateValue"

What each line does

• CWAgent disk_used_percent — The CloudWatch agent on the VM reports disk percent.
• path=/var/lib/jenkins — Watch the Jenkins disk, not the root disk.
• --period 300 --evaluation-periods 2 — Above the limit for 10 minutes.
• --threshold 80 — Warn at 80 percent, while there is still time.
• --alarm-actions ... sns — Send to the on-call email or Slack topic.

Green output

"OK"

Expected result. When the disk passes 80 percent for 10 minutes, on-call gets a message before builds break.

Verify: did it work?

1. Disk percent

   df -h /var/lib/jenkins | tail -1
   

   What it does: Below 70 is healthy.

   Output

   /dev/nvme1n1  100G  58G  42G  58% /var/lib/jenkins
   

2. Jobs without a discard rule

   grep -L "BuildDiscarderProperty" /var/lib/jenkins/jobs/*/config.xml | head
   

   What it does: Old freestyle jobs may need the rule set in the UI.

   Output

   /var/lib/jenkins/jobs/data-export/config.xml
   

3. Docker space on an agent

   docker system df
   

   What it does: Shows images, containers and cache sizes.

   Output

   TYPE         TOTAL  ACTIVE  SIZE    RECLAIMABLE
   Images       41     6       18.2GB  14.9GB (81%)
   

When it is red 1

java.io.IOException: No space left on device

Cause. The disk is 100 percent full.

Fix, step by step

1. Free space with the Script Console build delete.
2. Delete agent workspaces.
3. Grow the disk if needed.

When it is red 2

ERROR: Cannot delete workspace: Unable to delete ... Permission denied

Cause. Files in the workspace were made by root inside a Docker build.

Fix, step by step

1. Run docker run --rm -v "$WORKSPACE":/w alpine rm -rf /w/* first.
2. Use --user in docker agents so files belong to jenkins.

When it is red 3

df says 100 percent but du shows much less

Cause. A deleted file is still open by a running process, often a huge log.

Fix, step by step

1. sudo lsof +L1 | head
2. Restart that process to release the space.

When it is red 4

Disk fills again in a week

Cause. No discard rules and no agent cleanup.

Fix, step by step

1. Put buildDiscarder in the shared library.
2. Add the cron cleanup to every agent.

When it is red 5

ERROR: failed to solve: write /var/lib/docker/tmp/...: no space left on device

Cause. The agent Docker folder is full, not the Jenkins disk. Docker build cache grows silently.

Fix, step by step

1. docker builder prune -af --filter until=72h
2. docker system df to confirm.
3. Put /var/lib/docker on its own larger disk on busy agents.

When it is red 6

Script Console: groovy.lang.MissingMethodException: No signature of method: null.getBuilds()

Cause. getItemByFullName returned null, because the job is inside a folder.

Fix, step by step

1. Use the full name with the folder, like mobile/android-app.
2. Copy it from the job URL.

Cleanup

1. Remove the test alarm

   aws cloudwatch delete-alarms --alarm-names jenkins-disk-80
   

   What it does: Only in the lab.

   Output

   (no output)
   

2. Remove the cron file in the lab

   sudo rm /etc/cron.d/agent-cleanup
   

   What it does: Lab only.

   Output

   (no output)
   

Practice tasks

Homework: try at home

Recap

• Disks fill slowly from builds, artifacts, workspaces and Docker.
• Measure with df and du first.
• Delete builds through Jenkins, not with rm.
• buildDiscarder, cleanWs and Docker prune keep it clean.
• Alert at 80 percent.

FAQ

How many builds should I keep?

20 to 50 is common. Keep long history only for release jobs.

Is cleanWs bad for build speed?

A little. Use caches for dependencies, not old workspaces.

Can I move JENKINS_HOME to a bigger disk?

Yes: stop Jenkins, copy with rsync -a, update the mount, start.

Do Kubernetes agents need cleanup?

Less, because Pods are thrown away. But the node Docker or containerd cache still grows.

What uses the most space usually?

Archived artifacts like APKs, jars and test videos, then old build logs, then agent Docker images.

Can I compress old logs?

Jenkins does not compress build logs by default. Deleting old builds is simpler and safer than compressing.

Why use the Script Console carefully?

It runs with full admin power. Test on one job first.

Interview questions

1. Jenkins disk is full. What do you do?

   df and du to find the cause, delete old builds through Jenkins, clean workspaces, prune Docker on agents, then add discard rules and an 80 percent alert.

2. How do you stop old builds piling up?

   buildDiscarder with logRotator in every pipeline, ideally from a shared library.

3. Why might df and du disagree?

   Deleted files held open by a process; find with lsof +L1.

4. Where should large artifacts live?

   In S3, Nexus or Artifactory, not on the controller.

5. How do you clean agents?

   cleanWs after builds and a scheduled docker system prune with a time filter.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview cardComparison card

मराठी cards