Chapter 80: DevOps Projects Part 10: Jenkins Agents Freestyle Pipeline Webhook Sonar ECR EKS Approval Libraries Backup

80.10 Project 76: Jenkins backup and restore

By Ravindra Bagale Last updated: 28 October 2026

Friends, a lost JENKINS_HOME is a lost company. Today we stop briefly, archive JENKINS_HOME, copy to S3, practice restore on a lab controller, and document the runbook.

What you will build in this project

Locate JENKINS_HOME.
Create a consistent archive.
Upload to S3 with encryption.
Restore on a lab controller and verify jobs.

1. Start from a real company story

At IBM in Bengaluru, a disk failure wiped an unbacked Jenkins and three days of job history and credentials vanished.

Rebuild took a week. Rough cost: about ₹80 lakh in idle engineers and delayed releases.

Arun set nightly encrypted S3 backups with a quarterly restore drill.

Today we practice that drill.

Picture 1: Jenkins → tar JENKINS_HOME → S3 encrypted → lab restore → verify jobs

Ravindra Bagale's Tip

Students back up while jobs write and get a corrupt tar. Quiesce or use a filesystem snapshot. Keep that habit.

2. Start and prepare

Step 1: Why backup

Why: Controller disk fails. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 1: a1

Type this:

echo 'JENKINS_HOME holds jobs, credentials, plugins, build history'

You will see:

JENKINS_HOME holds jobs, credentials, plugins, build history

What it means: What.

Example 2: a2

Type this:

mkdir -p /tmp/jen76/{backup,restore,runbook} && ls /tmp/jen76

You will see:

backup\nrestore\nrunbook

What it means: Folders.

Example 3: a3

Type this:

echo 'Today: tar, S3, lab restore, runbook'

You will see:

Today: tar, S3, lab restore, runbook

What it means: Scope.

Step 2: Find home

Why: Default paths. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: locate. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 4: b1

Type this:

printf '%s\n' 'Debian package default: /var/lib/jenkins' 'Docker often: /var/jenkins_home' 'Confirm: echo $JENKINS_HOME on the controller' > /tmp/jen76/HOME.txt && cat /tmp/jen76/HOME.txt

You will see:

Debian package default: /var/lib/jenkins

What it means: Paths.

Example 5: b2

Type this:

echo 'du -sh /var/lib/jenkins'

You will see:

du -sh /var/lib/jenkins

What it means: Size.

Example 6: b3

Type this:

echo 'Exclude workspace caches if you need smaller nightly tars'

You will see:

Exclude workspace caches if you need smaller nightly tars

What it means: Exclude tip.

Step 3: Quiesce tip

Why: Consistent archive. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: quiet down. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 7: c1

Type this:

printf '%s\n' 'Option A: Jenkins quietDown then tar' 'Option B: EBS/LVM snapshot under the hood' 'Option C: thin backup plugins that stream jobs/' > /tmp/jen76/QUIESCE.txt && cat /tmp/jen76/QUIESCE.txt

You will see:

Option A: Jenkins quietDown then tar

What it means: Options.

Example 8: c2

Type this:

echo 'java -jar jenkins-cli.jar -s http://CONTROLLER:8080 -auth user:token quiet-down'

You will see:

java -jar jenkins-cli.jar -s http://CONTROLLER:8080 -auth user:token quiet-down

What it means: quiet-down.

Example 9: c3

Type this:

echo 'cancel-quiet-down after the tar finishes'

You will see:

cancel-quiet-down after the tar finishes

What it means: Cancel.

3. Build the main pieces

Step 4: Tar archive

Why: Create backup file. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: tar tip. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 10: d1

Type this:

echo 'sudo tar --exclude=workspace --exclude=caches -C /var/lib -czf /tmp/jen76/backup/jenkins-home.tgz jenkins'

You will see:

sudo tar --exclude=workspace --exclude=caches -C /var/lib -czf /tmp/jen76/backup/jenkins-home.tgz jenkins

What it means: Tar.

Example 11: d2

Type this:

printf '%s\n' 'DATE=$(date +%F)' 'NAME=jenkins-home-$DATE.tgz' > /tmp/jen76/backup/NAME.txt && cat /tmp/jen76/backup/NAME.txt

You will see:

NAME=jenkins-home-$DATE.tgz

What it means: Name.

Example 12: d3

Type this:

echo 'sha256sum the tar and keep the hash next to the object'

You will see:

sha256sum the tar and keep the hash next to the object

What it means: Hash.

Step 5: S3 upload

Why: Encrypted object. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: aws s3 cp. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 13: e1

Type this:

echo 'aws s3 cp /tmp/jen76/backup/jenkins-home.tgz s3://company-jenkins-backups/$(date +%F)/jenkins-home.tgz --sse AES256'

You will see:

aws s3 cp /tmp/jen76/backup/jenkins-home.tgz s3://company-jenkins-backups/$(date +%F)/jenkins-home.tgz --sse AES256

What it means: Upload.

Example 14: e2

Type this:

printf '%s\n' 'Bucket policy: deny public' 'Lifecycle: transition to Glacier after 30 days' 'Retention: 90 days minimum' > /tmp/jen76/backup/S3.txt && cat /tmp/jen76/backup/S3.txt

You will see:

Bucket policy: deny public

What it means: Policy.

Example 15: e3

Type this:

echo 'Prefer SSE-KMS with a dedicated key for regulated teams'

You will see:

Prefer SSE-KMS with a dedicated key for regulated teams

What it means: KMS.

Step 6: Restore lab

Why: Practice on a spare. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: restore steps. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 16: f1

Type this:

printf '%s\n' '1. Install same Jenkins LTS on lab' '2. Stop jenkins service' '3. Wipe empty JENKINS_HOME' '4. tar -xzf backup into place' '5. Fix ownership jenkins:jenkins' '6. Start jenkins' > /tmp/jen76/restore/STEPS.txt && cat /tmp/jen76/restore/STEPS.txt

You will see:

4. tar -xzf backup into place

What it means: Steps.

Example 17: f2

Type this:

echo 'sudo tar -C /var/lib -xzf /tmp/jen76/backup/jenkins-home.tgz'

You will see:

sudo tar -C /var/lib -xzf /tmp/jen76/backup/jenkins-home.tgz

What it means: Extract.

Example 18: f3

Type this:

echo 'sudo chown -R jenkins:jenkins /var/lib/jenkins'

You will see:

sudo chown -R jenkins:jenkins /var/lib/jenkins

What it means: Owner.

4. Automate and verify

Step 7: Verify restore

Why: Jobs and plugins. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: checks. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 19: g1

Type this:

printf '%s\n' 'Login as admin' 'Open a known freestyle job' 'Confirm agent credentials still listed' 'Run smoke-agent01 once' > /tmp/jen76/restore/VERIFY.txt && cat /tmp/jen76/restore/VERIFY.txt

You will see:

Run smoke-agent01 once

What it means: Verify.

Example 20: g2

Type this:

echo 'Plugin versions should match; update only after restore is proven'

You will see:

Plugin versions should match; update only after restore is proven

What it means: Plugins.

Example 21: g3

Type this:

printf '%s\n' 'Acceptance: smoke job blue on restored lab' > /tmp/jen76/ACCEPT.txt && cat /tmp/jen76/ACCEPT.txt

You will see:

Acceptance: smoke job blue on restored lab

What it means: Gate.

Step 8: Secrets after restore

Why: Rotate when needed. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: rotate tip. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 22: h1

Type this:

echo 'If the backup could have leaked, rotate all credentials after restore'

You will see:

If the backup could have leaked, rotate all credentials after restore

What it means: Rotate.

Example 23: h2

Type this:

echo 'Re-issue agent SSH keys if the old controller disk was stolen'

You will see:

Re-issue agent SSH keys if the old controller disk was stolen

What it means: Keys.

Example 24: h3

Type this:

echo 'Update forge webhook secrets if URLs changed'

You will see:

Update forge webhook secrets if URLs changed

What it means: Hooks.

Step 9: Runbook

Why: Write it down. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: runbook file. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 25: i1

Type this:

printf '%s\n' '# Jenkins backup runbook' 'Nightly: quietDown -> tar -> s3 cp -> cancel-quiet-down' 'Monthly: restore to lab and run smoke' 'Contacts: oncall-devops@example.com' > /tmp/jen76/runbook/RUNBOOK.md && cat /tmp/jen76/runbook/RUNBOOK.md

You will see:

# Jenkins backup runbook

What it means: Runbook.

Example 26: i2

Type this:

echo 'Store the runbook outside JENKINS_HOME too'

You will see:

Store the runbook outside JENKINS_HOME too

What it means: Where.

Example 27: i3

Type this:

echo 'Calendar: quarterly restore drill with a named owner'

You will see:

Calendar: quarterly restore drill with a named owner

What it means: Drill.

5. Harden and accept

Step 10: Story IBM

Why: Bengaluru disk loss. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 28: j1

Type this:

printf '%s\n' 'At IBM in Bengaluru, unbacked Jenkins disk failed and job history plus credentials vanished.' 'Arun set nightly encrypted S3 backups with a quarterly restore drill.' > /tmp/jen76/STORY.txt && wc -l /tmp/jen76/STORY.txt

You will see:

2 /tmp/jen76/STORY.txt

What it means: Story.

Example 29: j2

Type this:

echo 'Rough cost: about ₹80 lakh in idle engineers and delayed releases'

You will see:

Rough cost: about ₹80 lakh in idle engineers and delayed releases

What it means: Cost.

Example 30: j3

Type this:

echo 'Arun: PASS'

You will see:

Arun: PASS

What it means: Sign-off.

Step 11: Proof

Why: Artifacts. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 31: k1

Type this:

ls /tmp/jen76/backup /tmp/jen76/restore /tmp/jen76/runbook

You will see:

/tmp/jen76/backup:\nNAME.txt\nS3.txt\n\n/tmp/jen76/restore:\nSTEPS.txt\nVERIFY.txt\n\n/tmp/jen76/runbook:\nRUNBOOK.md

What it means: Tree.

Example 32: k2

Type this:

test -f /tmp/jen76/ACCEPT.txt && echo 'accept ok'

You will see:

accept ok

What it means: accept.

Example 33: k3

Type this:

echo 'Jenkins projects 67-76 complete'

You will see:

Jenkins projects 67-76 complete

What it means: Done block.

Step 12: Acceptance

Why: Backup and restore proven. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 34: z1

Type this:

echo 'Encrypted S3 object exists'

You will see:

Encrypted S3 object exists

What it means: S3.

Example 35: z2

Type this:

echo 'Lab restore smoke blue'

You will see:

Lab restore smoke blue

What it means: Restore.

Example 36: z3

Type this:

echo 'Runbook stored outside JENKINS_HOME'

You will see:

Runbook stored outside JENKINS_HOME

What it means: Runbook.

Check yourself

Q1. Who fixed IBM lost Jenkins?

Arun

Q2. Home dir?

JENKINS_HOME

Q3. Object store?

Q4. Encrypt?

yes

Q5. Prove with?

restore drill

Q6. Corrupt risk?

live writes

Friends, you backed up and restored Jenkins. Did you understand? Let us try it once more. Schedule a monthly restore drill calendar invite for your team.

Lab: try at home

Find JENKINS_HOME.
Make encrypted archive.
Upload S3.
Restore lab and verify.

Recap

Backup is useless until restore works.
Encrypt at rest.
Quiesce first.
Drill quarterly.
Secrets rotate after restore.

Lab

Finish all ten projects; keep ACCEPT notes for each.
Never commit sonar tokens, AWS keys, or hook secrets.
Pin shared library versions; never float on main for prod.
After each project, write one paragraph: what broke and how you fixed it.
Next batch starts GitHub Actions and GitLab CI projects 77-84.

Practice task

Pick the webhook drill and the prod approval input. Prove both before you call the chapter done.

Ravindra Bagale's Tip

Keep the controller thin. Pipeline as code. Webhooks beat polling. Gate on Sonar. SHA tags not latest. Approve prod. Pin shared libraries. Backup is useless until restore works.

In short

Thin controller.
Freestyle then pipeline.
Jenkinsfile in git.
Hooks beat polls.
Sonar gate blocks.
SHA to ECR.
Same SHA to EKS.
Human gate for prod.
Pin shared libs.
Restore drill monthly.