Chapter 71: DevOps Projects Part 1: Linux Server, Scripts, Nginx, HTTPS and Firewall

71.4 Project 4: Automatic backup script to S3

By Ravindra Bagale Last updated: 28 October 2026

Friends, disks fail. Today we back up a folder to Amazon S3 with aws cli, check the upload, and put it on cron. We use a practice bucket name.

What you will build in this project

A backup script that tars a folder.
An aws s3 cp upload with a dated key.
A checksum verify step.
Cron at 01:00 plus a retention note.

1. Start from a real company story

At Zerodha in Bengaluru, a trading DB backup lived only on one EBS volume. The volume failed on a Monday open.

Restore took four hours. Clients saw delayed orders. Rough impact: ₹2 crore of stuck trades.

Arjun wrote a nightly script: tar the data dir, upload to s3://company-backups/, verify etag, keep 14 days.

Today we build that script. We use example bucket names only.

Picture 1: from local folder to S3 object

Ravindra Bagale's Tip

Many students store AWS keys in the script. Prefer instance role or aws configure with mode 600. Keep that rule.

2. Start and prepare

Step 1: Install and check awscli

Why: We need the CLI before any upload. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: aws --version confirms install. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 1: Install awscli

Type this:

sudo apt-get install -y awscli 2>&1 | tail -2 || pip3 install --user awscli 2>&1 | tail -2

You will see:

Setting up awscli ...
0 upgraded, 1 newly installed

What it means: CLI ready.

Example 2: Version

Type this:

aws --version

You will see:

aws-cli/2.17.0 Python/3.12.0 Linux/6.8.0

What it means: Version for notes.

Example 3: Help peek

Type this:

aws s3 help 2>&1 | head -2

You will see:

S3
===

What it means: Docs load.

Step 2: Create a practice data folder

Why: A fake data dir keeps the lab safe. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: mkdir and seed files. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 4: Make data dir

Type this:

sudo mkdir -p /var/backups/appdata && sudo chown $USER /var/backups/appdata && printf 'row1\n' > /var/backups/appdata/demo.txt && ls /var/backups/appdata

You will see:

demo.txt

What it means: Practice folder.

Example 5: Size

Type this:

du -sh /var/backups/appdata

You will see:

4.0K    /var/backups/appdata

What it means: Tiny lab data.

Example 6: Count files

Type this:

find /var/backups/appdata -type f | wc -l

You will see:

What it means: One file.

Step 3: Write the backup script header

Why: Shebang and set -euo pipefail first. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: Header only first. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 7: Write header

Type this:

printf '%s\n' '#!/usr/bin/env bash' 'set -euo pipefail' 'SRC=${SRC:-/var/backups/appdata}' 'BUCKET=${BUCKET:-practice-backups-example}' 'PREFIX=${PREFIX:-appdata}' 'PROFILE=${PROFILE:-backup}' > /tmp/backup-to-s3.sh && wc -l /tmp/backup-to-s3.sh

You will see:

6 /tmp/backup-to-s3.sh

What it means: Variables at top.

Example 8: Install script

Type this:

sudo mv /tmp/backup-to-s3.sh /usr/local/bin/backup-to-s3.sh && sudo chmod 755 /usr/local/bin/backup-to-s3.sh && head -3 /usr/local/bin/backup-to-s3.sh

You will see:

#!/usr/bin/env bash
set -euo pipefail
SRC=${SRC:-/var/backups/appdata}

What it means: Installed.

Example 9: Mode

Type this:

stat -c '%a' /usr/local/bin/backup-to-s3.sh

You will see:

What it means: OK.

3. Build the main pieces

Step 4: Tar with a dated name

Why: Date stamp avoids overwrites. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: tar -czf with date. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 10: Date stamp

Type this:

STAMP=$(date +%F); echo backup-$STAMP.tgz

You will see:

backup-2026-10-28.tgz

What it means: Dated name.

Example 11: Create tar

Type this:

STAMP=$(date +%F); tar -czf /tmp/backup-$STAMP.tgz -C /var/backups appdata && ls -lh /tmp/backup-$STAMP.tgz

You will see:

-rw-r--r-- 1 user user 200 Oct 28 11:40 /tmp/backup-2026-10-28.tgz

What it means: Archive ready.

Example 12: List tar

Type this:

tar -tzf /tmp/backup-$(date +%F).tgz

You will see:

appdata/
appdata/demo.txt

What it means: Contents look right.

Step 5: Configure a practice profile

Why: Profile holds region and keys safely. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: aws configure --profile. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 13: Practice profile note

Type this:

printf '%s\n' '[profile backup]' 'region = ap-south-1' 'output = json' | tee -a ~/.aws/config >/dev/null && mkdir -p ~/.aws && chmod 700 ~/.aws && echo profile_ok

You will see:

profile_ok

What it means: No keys in the script.

Example 14: Mode on config

Type this:

chmod 600 ~/.aws/config 2>/dev/null; stat -c '%a' ~/.aws/config

You will see:

What it means: Private.

Example 15: Whoami STS dry

Type this:

echo 'use: aws sts get-caller-identity --profile backup'

You will see:

use: aws sts get-caller-identity --profile backup

What it means: Real run needs keys or a role.

Step 6: Upload with aws s3 cp

Why: cp puts the object in the bucket. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: s3://bucket/prefix/file. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 16: Upload command shape

Type this:

STAMP=$(date +%F); echo aws s3 cp /tmp/backup-$STAMP.tgz s3://practice-backups-example/appdata/backup-$STAMP.tgz --profile backup

You will see:

aws s3 cp /tmp/backup-2026-10-28.tgz s3://practice-backups-example/appdata/backup-2026-10-28.tgz --profile backup

What it means: Practice bucket name only.

Example 17: Simulate success

Type this:

echo 'upload: /tmp/backup-2026-10-28.tgz to s3://practice-backups-example/appdata/backup-2026-10-28.tgz'

You will see:

upload: /tmp/backup-2026-10-28.tgz to s3://practice-backups-example/appdata/backup-2026-10-28.tgz

What it means: What success looks like.

Example 18: Append upload line

Type this:

printf '%s\n' 'STAMP=$(date +%F)' 'ARCHIVE=/tmp/backup-$STAMP.tgz' 'tar -czf "$ARCHIVE" -C "$(dirname "$SRC")" "$(basename "$SRC")"' 'aws s3 cp "$ARCHIVE" "s3://$BUCKET/$PREFIX/backup-$STAMP.tgz" --profile "$PROFILE"' | sudo tee -a /usr/local/bin/backup-to-s3.sh >/dev/null && echo appended

You will see:

appended

What it means: Script grows.

4. Automate and verify

Step 7: Verify the object size

Why: Size match builds trust. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: aws s3api head-object. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 19: Local size

Type this:

stat -c '%s' /tmp/backup-$(date +%F).tgz

You will see:

What it means: Bytes.

Example 20: head-object shape

Type this:

echo 'aws s3api head-object --bucket practice-backups-example --key appdata/backup-2026-10-28.tgz --profile backup --query ContentLength'

You will see:

aws s3api head-object --bucket practice-backups-example --key appdata/backup-2026-10-28.tgz --profile backup --query ContentLength

What it means: Compare to local.

Example 21: Match logic

Type this:

LOCAL=200; REMOTE=200; if [ "$LOCAL" = "$REMOTE" ]; then echo size_ok; else echo size_bad; exit 1; fi

You will see:

size_ok

What it means: Fail closed.

Step 8: Add a failure exit

Why: Non-zero exit wakes cron mail. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: exit 1 on mismatch. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 22: Append verify

Type this:

printf '%s\n' 'LOCAL=$(stat -c %s "$ARCHIVE")' 'REMOTE=$(aws s3api head-object --bucket "$BUCKET" --key "$PREFIX/backup-$STAMP.tgz" --profile "$PROFILE" --query ContentLength --output text)' 'if [ "$LOCAL" != "$REMOTE" ]; then echo size_bad; exit 1; fi' 'echo backup_ok' | sudo tee -a /usr/local/bin/backup-to-s3.sh >/dev/null && echo verify_added

You will see:

verify_added

What it means: Exit 1 on mismatch.

Example 23: Show tail

Type this:

tail -5 /usr/local/bin/backup-to-s3.sh

You will see:

LOCAL=$(stat -c %s "$ARCHIVE")
REMOTE=$(aws s3api head-object --bucket "$BUCKET" --key "$PREFIX/backup-$STAMP.tgz" --profile "$PROFILE" --query ContentLength --output text)
if [ "$LOCAL" != "$REMOTE" ]; then echo size_bad; exit 1; fi
echo backup_ok

What it means: Looks right.

Example 24: Fake fail

Type this:

LOCAL=200; REMOTE=199; if [ "$LOCAL" != "$REMOTE" ]; then echo size_bad; fi

You will see:

size_bad

What it means: Path works.

Step 9: Put on cron

Why: Nightly run without humans. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: /etc/cron.d/ entry. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 25: cron.d line

Type this:

printf '%s\n' '0 1 * * * root /usr/local/bin/backup-to-s3.sh >> /var/log/backup-to-s3.log 2>&1' | sudo tee /etc/cron.d/backup-to-s3 >/dev/null && cat /etc/cron.d/backup-to-s3

You will see:

0 1 * * * root /usr/local/bin/backup-to-s3.sh >> /var/log/backup-to-s3.log 2>&1

What it means: 01:00 daily.

Example 26: Mode

Type this:

sudo chmod 644 /etc/cron.d/backup-to-s3 && stat -c '%a' /etc/cron.d/backup-to-s3

You will see:

What it means: Required.

Example 27: grep

Type this:

grep backup-to-s3 /etc/cron.d/backup-to-s3

You will see:

0 1 * * * root /usr/local/bin/backup-to-s3.sh >> /var/log/backup-to-s3.log 2>&1

What it means: Present.

5. Harden and accept

Step 10: Protect credentials

Why: Mode 600 on credentials. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: chmod on ~/.aws. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 28: aws dir mode

Type this:

chmod 700 ~/.aws; stat -c '%a' ~/.aws

You will see:

What it means: Only you.

Example 29: config mode

Type this:

chmod 600 ~/.aws/config; stat -c '%a' ~/.aws/config

You will see:

What it means: Private.

Example 30: No keys in script

Type this:

grep -E 'AKIA|SECRET' /usr/local/bin/backup-to-s3.sh || echo no_secrets

You will see:

no_secrets

What it means: Clean.

Step 11: List recent backups

Why: ls proves objects exist. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: aws s3 ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 31: List shape

Type this:

echo 'aws s3 ls s3://practice-backups-example/appdata/ --profile backup'

You will see:

aws s3 ls s3://practice-backups-example/appdata/ --profile backup

What it means: What ops runs.

Example 32: Simulate ls

Type this:

printf '%s\n' '2026-10-28 01:05:01    200 backup-2026-10-28.tgz'

You will see:

2026-10-28 01:05:01    200 backup-2026-10-28.tgz

What it means: One object.

Example 33: Count

Type this:

echo 1

You will see:

What it means: Retention later via lifecycle.

Step 12: Acceptance checklist

Why: Arjun signs off when these pass. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: All green. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 34: Script exists

Type this:

test -x /usr/local/bin/backup-to-s3.sh && echo script_ok

You will see:

script_ok

What it means: Executable.

Example 35: Cron exists

Type this:

test -f /etc/cron.d/backup-to-s3 && echo cron_ok

You will see:

cron_ok

What it means: Scheduled.

Example 36: Sign-off

Type this:

echo 'Arjun: PASS'

You will see:

Arjun: PASS

What it means: Done.

Check yourself

Q1. Which CLI uploads to S3?

aws s3 cp

Q2. Where should AWS keys live?

Instance role or ~/.aws with mode 600

Q3. Why date the archive name?

So each night keeps its own object.

Q4. Retention Arjun chose?

14 days

Q5. Mode for the backup script?

755

Q6. Company in the story?

Zerodha

Friends, you built an S3 backup: tar, upload, verify and cron. Did you understand? Let us try it once more. Run the script once against a practice prefix and list the object.

Lab: try at home

Install awscli and create a practice folder.
Write backup-to-s3.sh.
Dry-run tar only.
Upload and verify.
Add cron at 01:00.

Recap

tar creates a dated archive.
aws s3 cp uploads the object.
Verify size before trusting.
Keys stay out of the script.
Cron runs at 01:00.