84.2 Project 98: Disaster recovery across two AWS regions
Friends, one region is a single point of failure. Today we design RPO/RTO, replicate data, and run a failover drill.
मित्रांनो, एक region = single point of failure. आज RPO/RTO, replicate, failover drill.
दोस्तों, एक region = single point of failure. आज RPO/RTO, replicate, failover drill.
What you will build in this project
- Write RPO and RTO.
- Replicate data cross-region.
- Document failover DNS.
- Run a tabletop drill.
1. Start from a real company story
At Zerodha in Bengaluru, a region blip paused trading UI for customers who had no failover runbook.
Rough cost: trust damage harder than the Rs 20 lakh engineering overtime.
Ankit published RPO/RTO and a quarterly failover drill.
Today we write that drill.
Ravindra Bagale's Tip
Students claim zero RPO without sync replication budget. Be honest about RPO. Keep that habit.
Students sync budget नसताना zero RPO सांगतात. RPO प्रामाणिक ठेवा. सवय ठेवा.
Students sync budget के बिना zero RPO कहते हैं. RPO ईमानदार रखें. आदत रखिए.
2. Start and prepare
Step 1: Why this project
Why: Real outages teach. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 1: a1
Type this:
echo 'Today we build dr98'
You will see:
Today we build dr98
What it means: Why.
Example 2: a2
Type this:
mkdir -p /tmp/dr98 && echo 'Scope ready'
You will see:
Scope ready
What it means: Scope.
Example 3: a3
Type this:
echo 'Keep commands short and check every output'
You will see:
Keep commands short and check every output
What it means: Habit.
Step 2: Prepare workspace
Why: Folders and notes. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: prep. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 4: b1
Type this:
mkdir -p /tmp/dr98 && printf '%s\n' 'lab notes' > /tmp/dr98/NOTES.txt && cat /tmp/dr98/NOTES.txt
You will see:
lab notes
What it means: Notes.
Example 5: b2
Type this:
echo 'Work only under /tmp/dr98 for this project'
You will see:
Work only under /tmp/dr98 for this project
What it means: Path.
Example 6: b3
Type this:
echo 'Write ACCEPT.txt at the end'
You will see:
Write ACCEPT.txt at the end
What it means: Accept file.
Step 3: RPO and RTO
Why: Write numbers. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: slo. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 7: c1
Type this:
printf '%s\n' 'RPO: max data loss tolerated (example 5 minutes)' 'RTO: max time to recover (example 30 minutes)' > /tmp/dr98/RPO.txt && cat /tmp/dr98/RPO.txt
You will see:
RPO: max data loss tolerated (example 5 minutes)
What it means: RPO.
Example 8: c2
Type this:
echo 'Business owns the numbers; engineering designs to them'
You will see:
Business owns the numbers; engineering designs to them
What it means: Owners.
Example 9: c3
Type this:
echo 'Cheaper RPO means sync replication and higher cost'
You will see:
Cheaper RPO means sync replication and higher cost
What it means: Cost.
3. Build the main pieces
Step 4: Data replication
Why: Async first. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: data. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 10: d1
Type this:
printf '%s\n' 'RDS cross-region read replica or Aurora global' 'S3 CRR for object data' 'DynamoDB global tables when needed' > /tmp/dr98/DATA.txt && cat /tmp/dr98/DATA.txt
You will see:
RDS cross-region read replica or Aurora global
What it means: Data.
Example 11: d2
Type this:
echo 'Know whether failover promotes read replica automatically'
You will see:
Know whether failover promotes read replica automatically
What it means: Promote.
Example 12: d3
Type this:
echo 'Test restore; replication without restore drills is faith'
You will see:
Test restore; replication without restore drills is faith
What it means: Restore.
Step 5: Stateless compute
Why: Easy side. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: compute. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 13: e1
Type this:
printf '%s\n' 'Bake AMIs or use containers in both regions' 'Keep Terraform modules parameterized by region' > /tmp/dr98/COMPUTE.txt && cat /tmp/dr98/COMPUTE.txt
You will see:
Keep Terraform modules parameterized by region
What it means: Compute.
Example 14: e2
Type this:
echo 'Warm standby costs more than pilot light; pick deliberately'
You will see:
Warm standby costs more than pilot light; pick deliberately
What it means: Mode.
Example 15: e3
Type this:
printf '%s\n' 'Acceptance: secondary can serve /healthz after promote steps' > /tmp/dr98/ACCEPT.txt && cat /tmp/dr98/ACCEPT.txt
You will see:
Acceptance: secondary can serve /healthz after promote steps
What it means: Gate.
Step 6: DNS failover
Why: Route53. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: dns. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 16: f1
Type this:
echo 'Route53 failover record with health checks on primary'
You will see:
Route53 failover record with health checks on primary
What it means: DNS.
Example 17: f2
Type this:
echo 'TTL low enough for your RTO; not so low it hammers DNS'
You will see:
TTL low enough for your RTO; not so low it hammers DNS
What it means: TTL.
Example 18: f3
Type this:
echo 'Document who is allowed to flip manual failover'
You will see:
Document who is allowed to flip manual failover
What it means: Who.
4. Automate and verify
Step 7: Tabletop drill
Why: People. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: drill. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 19: g1
Type this:
printf '%s\n' 'Quarterly: declare primary down' 'Follow the runbook aloud' 'Time each step against RTO' > /tmp/dr98/DRILL.txt && cat /tmp/dr98/DRILL.txt
You will see:
Time each step against RTO
What it means: Drill.
Example 20: g2
Type this:
echo 'Capture gaps and fix the runbook the same week'
You will see:
Capture gaps and fix the runbook the same week
What it means: Fix.
Example 21: g3
Type this:
echo 'Include communications: status page and customer text'
You will see:
Include communications: status page and customer text
What it means: Comms.
Step 8: Chaos lightly
Why: Prove it. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: chaos. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 22: h1
Type this:
echo 'Game days beat paper drills when safe to run'
You will see:
Game days beat paper drills when safe to run
What it means: Game day.
Example 23: h2
Type this:
echo 'Start in staging with traffic mirrors'
You will see:
Start in staging with traffic mirrors
What it means: Staging.
Example 24: h3
Type this:
echo 'Never surprise prod without executive cover'
You will see:
Never surprise prod without executive cover
What it means: Cover.
Step 9: Backups still matter
Why: DR is not backup. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: backup. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 25: i1
Type this:
printf '%s\n' 'Regional DR does not replace ransomware backups' 'Keep immutable backups offline or vaulted' > /tmp/dr98/BAK.txt && cat /tmp/dr98/BAK.txt
You will see:
Regional DR does not replace ransomware backups
What it means: Backup.
Example 26: i2
Type this:
echo 'Test restore monthly'
You will see:
Test restore monthly
What it means: Monthly.
Example 27: i3
Type this:
echo 'Encrypt backups and control who can delete'
You will see:
Encrypt backups and control who can delete
What it means: Protect.
5. Harden and accept
Step 10: Story Zerodha
Why: Bengaluru lesson. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 28: j1
Type this:
printf '%s\n' 'At Zerodha in Bengaluru, a region blip had no failover runbook.' 'Ankit published RPO/RTO and a quarterly drill.' > /tmp/dr98/STORY.txt && wc -l /tmp/dr98/STORY.txt
You will see:
2 /tmp/dr98/STORY.txt
What it means: Story.
Example 29: j2
Type this:
echo 'Rough cost: trust damage plus about Rs 20 lakh overtime.'
You will see:
Rough cost: trust damage plus about Rs 20 lakh overtime.
What it means: Cost.
Example 30: j3
Type this:
echo 'Ankit: PASS'
You will see:
Ankit: PASS
What it means: Sign-off.
Step 11: Proof
Why: Files. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 31: k1
Type this:
ls /tmp/dr98
You will see:
NOTES.txt\nSTORY.txt\nACCEPT.txt
What it means: Tree.
Example 32: k2
Type this:
test -f /tmp/dr98/ACCEPT.txt && echo ACCEPT_OK
You will see:
ACCEPT_OK
What it means: Accept present.
Example 33: k3
Type this:
echo 'Next: Serverless Lambda project 99'
You will see:
Next: Serverless Lambda project 99
What it means: Bridge.
Step 12: Acceptance
Why: DR drill documented. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 34: z1
Type this:
echo 'RPO set'
You will see:
RPO set
What it means: A.
Example 35: z2
Type this:
echo 'Replica path'
You will see:
Replica path
What it means: B.
Example 36: z3
Type this:
echo 'Drill completed'
You will see:
Drill completed
What it means: C.
Check yourself
Q1. Who fixed Zerodha runbook?
Ankit
Q2. Targets?
RPO RTO
Q3. Drill?
quarterly
Q4. Zero RPO free?
no
Q5. DNS?
failover
Q6. Verify?
yes
Friends, you documented failover. Did you understand? Let us try it once more. Run the tabletop: primary down, who flips DNS, how do you verify.
मित्रांनो, तुम्ही failover document केले. Tabletop: primary down, DNS कोण पलटतो, verify कसे.
दोस्तों, आपने failover document किया. Tabletop: primary down, DNS कौन पलटता है, verify कैसे.
Lab: try at home
- RPO/RTO written.
- Replica exists.
- DNS plan.
- Drill notes.
Recap
- Honest RPO/RTO.
- Replicate deliberately.
- DNS failover.
- Drill quarterly.
- People matter.