Chapter 79: DevOps Projects Part 9: Ansible Nginx Roles Vault Patch Inventory Agents

79.4 Project 64: Patch servers with zero downtime

By Ravindra Bagale Last updated: 28 October 2026

Friends, patching all nodes at once drops the site. Today we drain a target, patch with serial one, wait for health, rejoin, and write the abort window.

What you will build in this project

Drain from load balancer.
Serial apt upgrade playbook.
Health gate before rejoin.
Maintenance window notes.

1. Start from a real company story

At Razorpay in Bengaluru, a Saturday apt upgrade hit all payment nodes together and health checks failed for twelve minutes.

Rough cost: about ₹1 crore in delayed settlements.

Meera wrote serial equals one with drain and rejoin.

Today we practice that zero-downtime patch.

Picture 1: LB drain → patch → health → rejoin → next host

Ravindra Bagale's Tip

Students reboot the whole ASG with one command to finish faster. Capacity disappears. Prefer one node at a time. Keep that habit.

2. Start and prepare

Step 1: Why zero downtime

Why: Patch windows still need capacity. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 1: a1

Type this:

echo 'Take a node out of the LB, patch, verify, put it back'

You will see:

Take a node out of the LB, patch, verify, put it back

What it means: Pattern.

Example 2: a2

Type this:

mkdir -p /tmp/ans64/playbooks && echo 'serial + health checks = safer nights'

You will see:

serial + health checks = safer nights

What it means: Idea.

Example 3: a3

Type this:

echo 'Today: drain, apt upgrade, reboot when needed, rejoin'

You will see:

Today: drain, apt upgrade, reboot when needed, rejoin

What it means: Scope.

Step 2: Inventory web

Why: Same ten nodes idea. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: hosts. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 4: b1

Type this:

printf '%s\n' '[web]' 'web01 ansible_host=10.0.1.11' 'web02 ansible_host=10.0.1.12' 'web03 ansible_host=10.0.1.13' '[web:vars]' 'ansible_user=ubuntu' > /tmp/ans64/hosts.ini && cat /tmp/ans64/hosts.ini

You will see:

web03 ansible_host=10.0.1.13

What it means: Hosts.

Example 5: b2

Type this:

echo 'Use the real ten-host file from project 61 in production drills'

You will see:

Use the real ten-host file from project 61 in production drills

What it means: Reuse.

Example 6: b3

Type this:

printf '%s\n' '[defaults]' 'inventory = ./hosts.ini' > /tmp/ans64/ansible.cfg && cat /tmp/ans64/ansible.cfg

You will see:

inventory = ./hosts.ini

What it means: cfg.

Step 3: Drain task

Why: Remove from target group. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: aws elbv2 tip. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 7: c1

Type this:

printf '%s\n' '---' '- name: Drain instance from TG' '  delegate_to: localhost' '  command: aws elbv2 deregister-targets --target-group-arn {{ tg_arn }} --targets Id={{ instance_id }}' > /tmp/ans64/playbooks/drain.yml && cat /tmp/ans64/playbooks/drain.yml

You will see:

deregister-targets

What it means: Drain.

Example 8: c2

Type this:

echo 'Wait for connection draining seconds before patching'

You will see:

Wait for connection draining seconds before patching

What it means: Wait.

Example 9: c3

Type this:

echo 'Store tg_arn and instance_id in host vars'

You will see:

Store tg_arn and instance_id in host vars

What it means: Vars.

3. Build the main pieces

Step 4: Patch playbook

Why: apt upgrade carefully. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: patch.yml. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 10: d1

Type this:

printf '%s\n' '---' '- name: Patch web tier' '  hosts: web' '  become: true' '  serial: 1' '  max_fail_percentage: 0' '  tasks:' '    - name: Update apt cache' '      apt: { update_cache: true, cache_valid_time: 3600 }' '    - name: Upgrade packages' '      apt: { upgrade: dist }' '    - name: Check if reboot required' '      stat: { path: /var/run/reboot-required }' '      register: reboot_required' '    - name: Reboot when needed' '      reboot:' '      when: reboot_required.stat.exists' > /tmp/ans64/playbooks/patch.yml && wc -l /tmp/ans64/playbooks/patch.yml

You will see:

18 /tmp/ans64/playbooks/patch.yml

What it means: Playbook.

Example 11: d2

Type this:

grep serial /tmp/ans64/playbooks/patch.yml

You will see:

serial: 1

What it means: One at a time.

Example 12: d3

Type this:

echo 'dist upgrade is stronger than safe — choose per policy'

You will see:

dist upgrade is stronger than safe — choose per policy

What it means: Policy tip.

Step 5: Health gate

Why: curl before rejoin. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: uri module. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 13: e1

Type this:

printf '%s\n' '- name: Wait for local health' '  uri:' '    url: http://127.0.0.1/health' '    status_code: 200' '  register: health' '  retries: 30' '  delay: 2' '  until: health.status == 200' > /tmp/ans64/playbooks/health-snippet.yml && cat /tmp/ans64/playbooks/health-snippet.yml

You will see:

status_code: 200

What it means: Health.

Example 14: e2

Type this:

echo 'Fail the host if health never returns — do not rejoin'

You will see:

Fail the host if health never returns — do not rejoin

What it means: Fail closed.

Example 15: e3

Type this:

echo 'Include /health in every app image before patch night'

You will see:

Include /health in every app image before patch night

What it means: App tip.

Step 6: Rejoin

Why: Register target again. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: register-targets. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 16: f1

Type this:

echo 'aws elbv2 register-targets --target-group-arn $TG --targets Id=$IID'

You will see:

aws elbv2 register-targets --target-group-arn $TG --targets Id=$IID

What it means: Rejoin.

Example 17: f2

Type this:

echo 'Wait until target state is healthy in describe-target-health'

You will see:

Wait until target state is healthy in describe-target-health

What it means: Wait healthy.

Example 18: f3

Type this:

echo 'Only then move serial to the next host'

You will see:

Only then move serial to the next host

What it means: Serial gate.

4. Automate and verify

Step 7: Full flow play

Why: Include order. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: site patch. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 19: g1

Type this:

printf '%s\n' 'Flow per host: drain -> patch -> health -> rejoin' > /tmp/ans64/FLOW.txt && cat /tmp/ans64/FLOW.txt

You will see:

Flow per host: drain -> patch -> health -> rejoin

What it means: Flow.

Example 20: g2

Type this:

echo 'ansible-playbook playbooks/patch.yml --limit web01'

You will see:

ansible-playbook playbooks/patch.yml --limit web01

What it means: Limit first.

Example 21: g3

Type this:

echo 'Expand to serial 1 across the group after one success'

You will see:

Expand to serial 1 across the group after one success

What it means: Expand.

Step 8: Maintenance window

Why: Calendar and chat. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: window notes. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 22: h1

Type this:

printf '%s\n' 'Window: Tue 01:00-03:00 IST' 'On-call: patch lead + app owner' 'Abort if error rate > 1 percent' > /tmp/ans64/WINDOW.txt && cat /tmp/ans64/WINDOW.txt

You will see:

Abort if error rate > 1 percent

What it means: Window.

Example 23: h2

Type this:

echo 'Post start and end messages in the ops channel'

You will see:

Post start and end messages in the ops channel

What it means: Chat.

Example 24: h3

Type this:

echo 'Keep a known-good AMI to rebuild a failed node'

You will see:

Keep a known-good AMI to rebuild a failed node

What it means: AMI tip.

Step 9: Story Razorpay

Why: All-at-once patch. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 25: i1

Type this:

printf '%s\n' 'At Razorpay in Bengaluru, a Saturday apt upgrade hit all payment nodes together. Health checks failed fleet-wide for twelve minutes.' 'Rough cost: about ₹1 crore in delayed settlements.' 'Meera wrote serial=1 with drain and rejoin.' > /tmp/ans64/STORY.txt && wc -l /tmp/ans64/STORY.txt

You will see:

3 /tmp/ans64/STORY.txt

What it means: Story.

Example 26: i2

Type this:

echo 'Meera: PASS'

You will see:

Meera: PASS

What it means: Sign-off.

Example 27: i3

Type this:

echo 'Never set serial to the full group size on payment tiers'

You will see:

Never set serial to the full group size on payment tiers

What it means: Rule.

5. Harden and accept

Step 10: Local proof

Why: Artifacts. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 28: j1

Type this:

ls /tmp/ans64/playbooks

You will see:

drain.yml\nhealth-snippet.yml\npatch.yml

What it means: Playbooks.

Example 29: j2

Type this:

grep max_fail /tmp/ans64/playbooks/patch.yml

You will see:

max_fail_percentage: 0

What it means: Fail percent.

Example 30: j3

Type this:

cat /tmp/ans64/FLOW.txt

You will see:

Flow per host: drain -> patch -> health -> rejoin

What it means: Flow.

Step 11: Acceptance

Why: Patch night ready. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 31: k1

Type this:

echo 'serial 1 + drain + health + rejoin documented'

You will see:

serial 1 + drain + health + rejoin documented

What it means: Checks.

Example 32: k2

Type this:

echo 'Abort rules in WINDOW.txt'

You will see:

Abort rules in WINDOW.txt

What it means: Window.

Example 33: k3

Type this:

echo 'Next: dynamic AWS inventory project 65'

You will see:

Next: dynamic AWS inventory project 65

What it means: Bridge.

Step 12: Bridge

Why: Inventory next. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: bridge. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 34: z1

Type this:

echo 'Static hosts.ini is fine for labs; AWS needs dynamic inventory next'

You will see:

Static hosts.ini is fine for labs; AWS needs dynamic inventory next

What it means: Why next.

Example 35: z2

Type this:

echo 'Keep FLOW.txt in the runbook wiki'

You will see:

Keep FLOW.txt in the runbook wiki

What it means: Habit.

Example 36: z3

Type this:

echo 'Meera: PASS'

You will see:

Meera: PASS

What it means: Done.

Check yourself

Q1. Who fixed Razorpay patch outage?

Meera

Q2. Serial value?

Q3. Before patch?

drain

Q4. Before rejoin?

health

Q5. Fail percentage?

Q6. All-at-once OK?

Friends, you wrote a zero-downtime patch flow. Did you understand? Let us try it once more. Add a step that snapshots the root volume before reboot on database hosts.

Lab: try at home

Write drain and patch playbooks.
Add health gate.
Document window and abort.
Limit to one host first.

Recap

Drain before patch.
Serial protects capacity.
Health before rejoin.
Abort on error spike.
Keep a known-good AMI.