Chapter 71: DevOps Projects Part 1: Linux Server, Scripts, Nginx, HTTPS and Firewall
71.2 Project 2: Disk-usage alert script with email
Friends, disks fill up silently. Today we write a Bash script that checks disk use and emails on-call when a limit is crossed. We dry-run first, then add cron.
मित्रांनो, disk शांतपणे भरतो. आज Bash script लिहू जो disk use तपासून limit ओलांडल्यावर on-call ला email करेल. आधी dry-run, मग cron.
दोस्तों, disk चुपचाप भर जाता है. आज Bash script लिखेंगे जो disk use जाँचकर limit पार होने पर on-call को email करे. पहले dry-run, फिर cron.
What you will build in this project
- A script that reads df and finds mounts over a limit.
- An alert body with hostname and top folders.
- Dry-run mode.
- Cron every 15 minutes plus logrotate.
1. Start from a real company story
At PhonePe in Bengaluru, a payment log partition hit 100 percent at 2 AM. Receipts failed and tickets jumped to 1,200 in one hour.
A debug flag had been left on for three days. Logs grew by 40 GB. There was no disk alert.
Kabir wrote a short Bash script the next morning. Every 15 minutes it checks filesystems and emails on-call above 80 percent.
Outage cost about ₹6.5 lakh. The script took one hour. Today we build it.
Ravindra Bagale's Tip
Many students parse df with fragile cut columns. Prefer df -P and awk. Keep that habit.
खूप students df ला fragile cut ने parse करतात. df -P आणि awk वापरा. सवय ठेवा.
बहुत students df को fragile cut से parse करते हैं. df -P और awk इस्तेमाल करें. आदत रखिए.
2. Start and prepare
Step 1: Read disk use with df
Why: We need a number to compare. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: df shows use. -P is portable for scripts. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 1: Show filesystems
Type this:
df -P
You will see:
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/root 51200000 38000000 11000000 78% /
What it means: Root is at 78 percent.
Example 2: Skip memory filesystems
Type this:
df -P -x tmpfs -x devtmpfs
You will see:
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/root 51200000 38000000 11000000 78% /
What it means: tmpfs is RAM, skip it.
Example 3: Print mount and percent
Type this:
df -P -x tmpfs -x devtmpfs | awk 'NR>1 {print $6, $5}'
You will see:
/ 78%
/boot 22%
What it means: $6 is mount, $5 is percent.
Step 2: Compare to a limit
Why: Decide ALERT or OK. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: Strip percent and compare to LIMIT. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 4: Strip percent
Type this:
echo 78% | tr -d '%'
You will see:
78
What it means: Plain number.
Example 5: List mounts over 80
Type this:
df -P -x tmpfs -x devtmpfs | awk 'NR>1 {gsub(/%/,"",$5); if($5+0>=80) print $6,$5}'
You will see:
(no output when healthy)
What it means: No line means healthy.
Example 6: Fake a high mount
Type this:
printf '%s\n' '/dev/sim 100 90 10 91% /data' | awk '{gsub(/%/,"",$5); if($5+0>=80) print $6,$5}'
You will see:
/data 91
What it means: Rule works without filling a disk.
Step 3: Script header
Why: Variables at the top. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: Shebang, set -euo pipefail, LIMIT, EMAIL, DRY_RUN. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 7: Create header
Type this:
printf '%s\n' '#!/usr/bin/env bash' 'set -euo pipefail' 'LIMIT=${LIMIT:-80}' 'EMAIL=${EMAIL:-oncall@example.com}' 'DRY_RUN=${DRY_RUN:-0}' > /tmp/disk-alert.sh && wc -l /tmp/disk-alert.sh
You will see:
5 /tmp/disk-alert.sh
What it means: example.com is practice only.
Example 8: Install binary
Type this:
sudo mv /tmp/disk-alert.sh /usr/local/bin/disk-alert.sh && sudo chmod 755 /usr/local/bin/disk-alert.sh && ls -l /usr/local/bin/disk-alert.sh
You will see:
-rwxr-xr-x 1 root root 120 Oct 28 11:00 /usr/local/bin/disk-alert.sh
What it means: Not world-writable.
Example 9: Override LIMIT
Type this:
LIMIT=75 bash -c 'echo $LIMIT'
You will see:
75
What it means: Cron will pass LIMIT the same way.
3. Build the main pieces
Step 4: Build the alert body
Why: On-call needs host, percent, folders. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: Hostname, time, then du. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 10: Host and time
Type this:
printf 'Host: %s\nWhen: %s\n' "$(hostname -f)" "$(date -Is)"
You will see:
Host: prod-pay-03.internal
When: 2026-10-28T02:15:01+05:30
What it means: First lines of every alert.
Example 11: Top folders
Type this:
du -x -h /var 2>/dev/null | sort -hr | head -5
You will see:
12G /var/log
3.1G /var/lib/docker
800M /var/cache
What it means: -x stays on one filesystem.
Example 12: Temp body file
Type this:
BODY=$(mktemp); printf 'ALERT\n' > "$BODY"; wc -l "$BODY"; rm -f "$BODY"
You will see:
1
What it means: mktemp avoids clashes.
Step 5: Dry-run versus email
Why: Never surprise the mailbox while testing. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: DRY_RUN prints; else mail sends. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 13: Dry-run path
Type this:
DRY_RUN=1; printf 'ALERT / 91%%\n' > /tmp/t.txt; if [ "$DRY_RUN" = 1 ]; then echo DRY; cat /tmp/t.txt; else echo SEND; fi
You will see:
DRY
ALERT / 91%
What it means: Safe for demos.
Example 14: Subject with host
Type this:
printf -v SUB 'disk ALERT %s %s%%' "$(hostname -s)" 91; echo "$SUB"
You will see:
disk ALERT prod-pay-03 91%
What it means: Easy to filter.
Example 15: Practice local mail
Type this:
printf 'practice\n' | mail -s 'disk-alert practice' "$USER" && echo sent
You will see:
sent
What it means: Install mailutils if needed.
Step 6: Assemble and run
Why: Put the loop together. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: Quiet when healthy; ALERT when forced. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 16: Append quiet exit
Type this:
printf '%s\n' 'rows=test' 'if [ -z "$rows" ]; then exit 0; fi' 'echo ALERT' >> /usr/local/bin/disk-alert.sh && tail -3 /usr/local/bin/disk-alert.sh
You will see:
rows=test
if [ -z "$rows" ]; then exit 0; fi
echo ALERT
What it means: Empty means exit 0.
Example 17: Healthy run
Type this:
sudo LIMIT=80 /usr/local/bin/disk-alert.sh; echo exit:$?
You will see:
exit:0
What it means: Quiet success.
Example 18: Forced alert
Type this:
sudo LIMIT=1 /usr/local/bin/disk-alert.sh | head -5
You will see:
ALERT
/ 78
What it means: LIMIT=1 forces the path.
4. Automate and verify
Step 7: Find what filled the disk
Why: Alerts need a culprit. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: du and find show big folders and files. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 19: Biggest directories
Type this:
sudo du -x -h / 2>/dev/null | sort -hr | head -5
You will see:
45G /
12G /var
12G /var/log
What it means: /var/log is common.
Example 20: Largest log files
Type this:
sudo find /var/log -type f -printf '%s %p\n' | sort -nr | head -3
You will see:
5368709120 /var/log/app/debug.log
104857600 /var/log/syslog.1
What it means: One debug.log is 5 GB.
Example 21: Deleted but open
Type this:
sudo lsof +L1 2>/dev/null | head -3 || echo none
You will see:
none
What it means: These hold space until restart.
Step 8: Install mail tools
Why: Script needs a sender. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: mailutils on Ubuntu. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 22: Install
Type this:
sudo apt-get install -y mailutils 2>&1 | tail -2
You will see:
Setting up mailutils ...
0 upgraded, 1 newly installed
What it means: Provides mail.
Example 23: mail.rc
Type this:
printf '%s\n' 'set sendmail="/usr/sbin/sendmail -F disk-alert"' | sudo tee /etc/mail.rc >/dev/null && cat /etc/mail.rc
You will see:
set sendmail="/usr/sbin/sendmail -F disk-alert"
What it means: From name is disk-alert.
Example 24: Verify binary
Type this:
command -v mail
You will see:
/usr/bin/mail
What it means: Ready.
Step 9: Put it on cron
Why: Humans forget; cron does not. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: Every 15 minutes with a log. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 25: Show crontab
Type this:
crontab -l 2>/dev/null || echo 'no crontab'
You will see:
no crontab
What it means: Fresh start.
Example 26: Install line
Type this:
(crontab -l 2>/dev/null; printf '%s\n' '*/15 * * * * LIMIT=80 /usr/local/bin/disk-alert.sh >> /var/log/disk-alert.log 2>&1') | crontab - && crontab -l
You will see:
*/15 * * * * LIMIT=80 /usr/local/bin/disk-alert.sh >> /var/log/disk-alert.log 2>&1
What it means: Every 15 minutes.
Example 27: Confirm schedule
Type this:
crontab -l | grep disk-alert
You will see:
*/15 * * * * LIMIT=80 /usr/local/bin/disk-alert.sh >> /var/log/disk-alert.log 2>&1
What it means: Scheduled.
5. Harden and accept
Step 10: Logrotate the alert log
Why: Alert logs grow too. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: Weekly rotate, keep four. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 28: Write rule
Type this:
printf '%s\n' '/var/log/disk-alert.log {' ' weekly' ' rotate 4' ' missingok' ' notifempty' ' compress' '}' | sudo tee /etc/logrotate.d/disk-alert && echo ok
You will see:
ok
What it means: Four weeks.
Example 29: Debug rule
Type this:
sudo logrotate -d /etc/logrotate.d/disk-alert 2>&1 | tail -3
You will see:
reading config ...
Handling 1 logs
What it means: Valid.
Example 30: Force once
Type this:
sudo touch /var/log/disk-alert.log && sudo logrotate -f /etc/logrotate.d/disk-alert && ls /var/log/disk-alert*
You will see:
/var/log/disk-alert.log
/var/log/disk-alert.log.1
What it means: Works.
Step 11: Protect the env file
Why: No secrets in the script body. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: Mode 600 env file. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 31: Create env
Type this:
printf '%s\n' 'LIMIT=80' 'EMAIL=oncall@example.com' | sudo tee /etc/disk-alert.env >/dev/null && sudo chmod 600 /etc/disk-alert.env && stat -c '%a %n' /etc/disk-alert.env
You will see:
600 /etc/disk-alert.env
What it means: Only root reads.
Example 32: Cron sources env
Type this:
printf '%s\n' '*/15 * * * * set -a; . /etc/disk-alert.env; set +a; /usr/local/bin/disk-alert.sh >> /var/log/disk-alert.log 2>&1' | crontab - && crontab -l | tail -1
You will see:
*/15 * * * * set -a; . /etc/disk-alert.env; set +a; /usr/local/bin/disk-alert.sh >> /var/log/disk-alert.log 2>&1
What it means: set -a exports vars.
Example 33: Confirm mode
Type this:
stat -c '%a' /etc/disk-alert.env
You will see:
600
What it means: Fix before real addresses.
Step 12: Acceptance checklist
Why: Kabir signs off only when these pass. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: Healthy, forced alert, cron, logrotate. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 34: Healthy exit 0
Type this:
sudo LIMIT=80 /usr/local/bin/disk-alert.sh; echo $?
You will see:
0
What it means: Quiet.
Example 35: Forced ALERT
Type this:
sudo LIMIT=1 /usr/local/bin/disk-alert.sh | grep -c ALERT
You will see:
1
What it means: Alert path works.
Example 36: Automation present
Type this:
crontab -l | grep -c disk-alert; test -f /etc/logrotate.d/disk-alert && echo logrotate_ok
You will see:
1
logrotate_ok
What it means: Done.
Check yourself
Q1. Which df flag is portable?
-P
Q2. What does DRY_RUN=1 do?
Prints the alert without sending email.
Q3. How often did Kabir schedule the check?
Every 15 minutes.
Q4. Why include du output?
So on-call sees the large folder without logging in.
Q5. Mode for /etc/disk-alert.env?
600
Q6. Which tool rotates the log?
logrotate
Friends, you built a disk alert companies actually run: df -P, limit, dry-run, email and cron. Did you understand? Let us try it once more. Set LIMIT=1 on a practice box and read the full alert body.
मित्रांनो, तुम्ही company सारखा disk alert बनवला. Practice box वर LIMIT=1 ठेवून alert body वाचा.
दोस्तों, आपने company जैसा disk alert बनाया. Practice box पर LIMIT=1 रखकर alert body पढ़ें.
Lab: try at home
- Write disk-alert.sh with LIMIT and DRY_RUN.
- Dry-run on a healthy disk.
- Force LIMIT=1 and confirm ALERT.
- Install cron every 15 minutes.
- Add logrotate for the alert log.
Recap
- df -P gives portable columns.
- Alert includes hostname and percent.
- Dry-run before real email.
- cron every 15 minutes.
- du shows which folder grew.