Chapter 31: Backup, Configuration as Code and Jenkins Operations

31.1 Backup JENKINS_HOME

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we run Jenkins for the Swiggy delivery team. Two hundred jobs, credentials, build history and plugin settings live in one folder called JENKINS_HOME. One night the disk fails. Without a backup, the team spends two weeks rebuilding jobs from memory, and nobody can deploy. A backup is a copy of that folder, taken every night, stored somewhere else, and checked.

What we will build

A nightly backup of JENKINS_HOME for three kinds of install: Jenkins on a Linux VM (bash + cron), Jenkins in Docker (volume backup), and Jenkins in Kubernetes (CronJob to a second PVC). Each one skips big folders you do not need, such as workspace.

What: every term in plain English

• JENKINS_HOME — The folder that holds all Jenkins data: config.xml, jobs, users, plugins, secrets.
• Backup — A copy of data kept somewhere else so you can restore it.
• Exclude — Folders you skip, such as workspace and caches, because Jenkins can rebuild them.
• tar.gz — One compressed file that holds a whole folder.
• Checksum — A sha256 fingerprint to prove the file did not change.
• Retention — How long you keep old backups, such as 14 days.
• Off-site copy — A copy on another machine or in S3, not on the same disk.

How: step by step

1. Find JENKINS_HOME

   sudo systemctl show jenkins -p Environment | tr " " "\n" | grep JENKINS_HOME
   

   What it does: Shows the folder path on a VM install.

   Output

   JENKINS_HOME=/var/lib/jenkins
   

2. See what is big

   sudo du -sh /var/lib/jenkins/* | sort -h | tail -5
   

   What it does: Workspace is often the biggest, and you can skip it.

   Output

   1.2G    /var/lib/jenkins/plugins
   3.8G    /var/lib/jenkins/jobs
   9.4G    /var/lib/jenkins/workspace
   

3. Install the script

   sudo mkdir -p /opt/jenkins-backup && sudo nano /opt/jenkins-backup/backup.sh   # paste Example 1
   

   What it does: The backup script lives in one known place.

   Output

   (saved)
   

4. Run it once by hand

   sudo bash /opt/jenkins-backup/backup.sh
   

   What it does: Always run by hand before you schedule it.

   Output

   backup ok: /backup/jenkins/jenkins-home-2026-10-25-0130.tgz
   

5. Check the file

   ls -lh /backup/jenkins/ && sha256sum -c /backup/jenkins/*.sha256
   

   What it does: File exists and checksum matches.

   Output

   -rw-r--r-- 1 root root 1.9G jenkins-home-2026-10-25-0130.tgz
   jenkins-home-2026-10-25-0130.tgz: OK
   

6. Schedule it nightly

   echo "30 1 * * * root bash /opt/jenkins-backup/backup.sh >> /var/log/jenkins-backup.log 2>&1" | sudo tee /etc/cron.d/jenkins-backup
   

   What it does: Runs at 01:30 every night when builds are quiet.

   Output

   30 1 * * * root bash /opt/jenkins-backup/backup.sh >> /var/log/jenkins-backup.log 2>&1
   

7. Copy it off the server

   aws s3 cp /backup/jenkins/ s3://shipit-jenkins-backups/ --recursive --exclude "*" --include "*$(date +%F)*"
   

   What it does: A backup on the same disk dies with the disk.

   Output

   upload: backup/jenkins/jenkins-home-2026-10-25-0130.tgz to s3://shipit-jenkins-backups/jenkins-home-2026-10-25-0130.tgz
   

Example 1: Linux VM: bash script and cron

Stack. Ubuntu 22.04, Jenkins LTS (apt), bash, cron

Goal. Linux VM: bash script and cron. The simplest backup for Jenkins installed with apt or yum.

backup.sh

#!/usr/bin/env bash
# /opt/jenkins-backup/backup.sh - nightly JENKINS_HOME backup (VM install)
set -euo pipefail
JH=/var/lib/jenkins
DEST=/backup/jenkins
STAMP=$(date +%F-%H%M)
mkdir -p "$DEST"
tar --exclude="$JH/workspace" --exclude="$JH/caches" \
    --exclude="$JH/war" --exclude="$JH/.cache" \
    -czf "$DEST/jenkins-home-$STAMP.tgz" -C "$(dirname $JH)" "$(basename $JH)"
sha256sum "$DEST/jenkins-home-$STAMP.tgz" > "$DEST/jenkins-home-$STAMP.tgz.sha256"
find "$DEST" -name "jenkins-home-*.tgz*" -mtime +14 -delete
echo "backup ok: $DEST/jenkins-home-$STAMP.tgz"

What each line does

• set -euo pipefail — Stop at the first error, so a half backup is not reported as ok.
• --exclude=$JH/workspace — Skip build workspaces; jobs rebuild them.
• tar -czf — Make one compressed file.
• sha256sum — Save a fingerprint for later checks.
• find ... -mtime +14 -delete — Keep 14 days, delete older backups.

Green output

$ sudo bash /opt/jenkins-backup/backup.sh
backup ok: /backup/jenkins/jenkins-home-2026-10-25-0130.tgz
$ sha256sum -c /backup/jenkins/jenkins-home-2026-10-25-0130.tgz.sha256
jenkins-home-2026-10-25-0130.tgz: OK

Expected result. A dated, checked backup file; older than 14 days are removed.

Example 2: Docker: back up the jenkins_home volume with a throwaway container

Stack. Docker 27, jenkins/jenkins:lts-jdk17, alpine

Goal. Docker: back up the jenkins_home volume with a throwaway container. Works without stopping Jenkins for a lab; for production stop or use quiet-down first.

backup-docker.sh

# backup-docker.sh - Jenkins running in Docker with volume jenkins_home
set -euo pipefail
STAMP=$(date +%F-%H%M)
docker run --rm \
  -v jenkins_home:/data:ro \
  -v "$PWD/backups":/out \
  alpine:3.20 \
  tar --exclude=/data/workspace --exclude=/data/caches -czf /out/jenkins-home-$STAMP.tgz -C /data .
ls -lh backups/

What each line does

• -v jenkins_home:/data:ro — Mount the Jenkins volume read-only, so the backup cannot change it.
• -v $PWD/backups:/out — Write the archive to a folder on the host.
• alpine:3.20 — A tiny image that has tar.
• --rm — Remove the helper container when done.

Green output

$ bash backup-docker.sh
total 1.1G
-rw-r--r-- 1 root root 1.1G Oct 25 01:31 jenkins-home-2026-10-25-0131.tgz

Expected result. A tgz of the volume in ./backups.

Example 3: Kubernetes: a CronJob that tars the Jenkins PVC into a second PVC every night

Stack. Kubernetes (kind or EKS), Jenkins Helm chart, CronJob

Goal. Kubernetes: a CronJob that tars the Jenkins PVC into a second PVC every night.

k8s/jenkins-backup-cronjob.yaml

# k8s/jenkins-backup-cronjob.yaml - Jenkins in Kubernetes with PVC jenkins-home
apiVersion: batch/v1
kind: CronJob
metadata:
  name: jenkins-backup
  namespace: jenkins
spec:
  schedule: "30 1 * * *"          # 01:30 every night
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: alpine:3.20
              command: ["sh", "-c"]
              args:
                - tar --exclude=./workspace --exclude=./caches -czf /backup/jenkins-$(date +%F).tgz -C /var/jenkins_home . && ls -lh /backup
              volumeMounts:
                - { name: home, mountPath: /var/jenkins_home, readOnly: true }
                - { name: backup, mountPath: /backup }
          volumes:
            - name: home
              persistentVolumeClaim: { claimName: jenkins-home }
            - name: backup
              persistentVolumeClaim: { claimName: jenkins-backup }

What each line does

• schedule: "30 1 * * *" — Cron time: 01:30 daily.
• concurrencyPolicy: Forbid — Never run two backups at once.
• readOnly: true — The backup Pod cannot change Jenkins data.
• claimName: jenkins-backup — A separate volume for archives.

Green output

$ kubectl -n jenkins create job --from=cronjob/jenkins-backup backup-now
job.batch/backup-now created
$ kubectl -n jenkins logs job/backup-now
-rw-r--r--    1 root     root      1.3G Oct 25 01:30 jenkins-2026-10-25.tgz

Expected result. A nightly archive in the backup PVC; you can trigger one now with create job --from.

Verify: did it work?

1. Check last night ran

   grep "backup ok" /var/log/jenkins-backup.log | tail -1
   

   What it does: The log shows a success line.

   Output

   backup ok: /backup/jenkins/jenkins-home-2026-10-25-0130.tgz
   

2. Check the archive opens

   tar -tzf /backup/jenkins/jenkins-home-2026-10-25-0130.tgz | grep -c config.xml
   

   What it does: Counts config.xml files inside. One per job plus the main one.

   Output

   214
   

3. Check the off-site copy

   aws s3 ls s3://shipit-jenkins-backups/ | tail -1
   

   What it does: The newest file is in S3.

   Output

   2026-10-25 01:33:10 2040109832 jenkins-home-2026-10-25-0130.tgz
   

When it is red 1

tar: /var/lib/jenkins/jobs/api/builds/57/log: file changed as we read it

Cause. A build was writing while tar read it.

Fix, step by step

1. Run backup at a quiet time.
2. Or put Jenkins in quiet-down mode first (Manage Jenkins > Prepare for Shutdown).
3. Or exclude builds/*/log if history is not needed.

When it is red 2

tar: /backup/jenkins: Cannot write: No space left on device

Cause. The backup disk is full.

Fix, step by step

1. Run df -h /backup.
2. Lower retention or exclude workspace.
3. Move old archives to S3.

When it is red 3

upload failed: ... An error occurred (AccessDenied) when calling the PutObject operation

Cause. The server role cannot write to the S3 bucket.

Fix, step by step

1. Check the instance role policy allows s3:PutObject on the bucket.
2. Do not put access keys in the script; use the instance role.

Cleanup

1. Remove a test archive

   rm /backup/jenkins/jenkins-home-test.tgz
   

   What it does: Delete practice files only.

   Output

   (no output)
   

2. Remove the Docker helper output

   rm -r backups/
   

   What it does: Lab only.

   Output

   (no output)
   

3. Delete the test job in Kubernetes

   kubectl -n jenkins delete job backup-now
   

   What it does: The CronJob stays.

   Output

   job.batch "backup-now" deleted
   

Practice tasks

Homework: try at home

Recap

• All Jenkins state lives in JENKINS_HOME.
• Back it up every night, skip workspace and caches.
• Save a checksum and keep an off-site copy.
• Run the script by hand before you schedule it.
• The same idea works on VM, Docker and Kubernetes.

FAQ

Do I need to stop Jenkins?

For a perfect copy, use quiet-down or stop. Many teams back up live at night and accept small risk.

Is the secrets folder needed?

Yes. Without secrets/ and secret.key, saved credentials cannot be decrypted.

Is a disk snapshot enough?

Snapshots help, but also keep a file backup you can restore anywhere.

Should I use a backup plugin?

Plugins exist, but a simple script is easy to read and test.

How big is a backup?

Usually a few GB once workspace is excluded.

Interview questions

1. What do you back up in Jenkins?

   JENKINS_HOME: config.xml, jobs, users, plugins, nodes, secrets; exclude workspace and caches.

2. Why include secrets?

   They hold the keys that decrypt stored credentials.

3. How do you schedule backups?

   cron on a VM, a Docker helper, or a Kubernetes CronJob.

4. How do you protect backups?

   Off-site copy, encryption, limited access, retention.

5. How do you know a backup works?

   Only by restoring it in a test.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview cardComparison card

मराठी cards