83.2 Project 92: Alerts with Alertmanager to Slack/email
Friends, a dashboard nobody watches is decoration. Today we fire a Prometheus alert and route it through Alertmanager to a webhook.
मित्रांनो, कोणी न पाहिलेले dashboard सजावट आहे. आज Prometheus alert, Alertmanager, webhook.
दोस्तों, कोई न देखे dashboard सजावट है. आज Prometheus alert, Alertmanager, webhook.
What you will build in this project
- Write an alerting rule.
- Run Alertmanager.
- Route to a webhook.
- Silence and prove it.
1. Start from a real company story
At Paytm in Noida, a disk filled overnight and paging went only to a dead email alias.
Rough cost: about Rs 18 lakh in morning incident overtime.
Farhan routed Alertmanager to a live on-call webhook with silences.
Today we wire that path.
Ravindra Bagale's Tip
Students page on every blip. Add for and pending, and route by severity. Keep that habit.
Students प्रत्येक blip वर page करतात. for/pending आणि severity route. सवय ठेवा.
Students हर blip पर page करते हैं. for/pending और severity route. आदत रखिए.
2. Start and prepare
Step 1: Why this project
Why: Real outages teach. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 1: a1
Type this:
echo 'Today we build am92'
You will see:
Today we build am92
What it means: Why.
Example 2: a2
Type this:
mkdir -p /tmp/am92 && echo 'Scope ready'
You will see:
Scope ready
What it means: Scope.
Example 3: a3
Type this:
echo 'Keep commands short and check every output'
You will see:
Keep commands short and check every output
What it means: Habit.
Step 2: Prepare workspace
Why: Folders and notes. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: prep. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 4: b1
Type this:
mkdir -p /tmp/am92 && printf '%s\n' 'lab notes' > /tmp/am92/NOTES.txt && cat /tmp/am92/NOTES.txt
You will see:
lab notes
What it means: Notes.
Example 5: b2
Type this:
echo 'Work only under /tmp/am92 for this project'
You will see:
Work only under /tmp/am92 for this project
What it means: Path.
Example 6: b3
Type this:
echo 'Write ACCEPT.txt at the end'
You will see:
Write ACCEPT.txt at the end
What it means: Accept file.
Step 3: Alerting rule
Why: Symptom based. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: rule. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 7: c1
Type this:
printf '%s\n' 'groups:' '- name: shop' ' rules:' ' - alert: HighErrorRate' ' expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05' ' for: 2m' ' labels: {severity: page}' ' annotations: {summary: High 5xx rate}' > /tmp/am92/rules.yml && cat /tmp/am92/rules.yml
You will see:
for: 2m
What it means: Rule.
Example 8: c2
Type this:
echo 'Load rules in Prometheus rule_files'
You will see:
Load rules in Prometheus rule_files
What it means: Load.
Example 9: c3
Type this:
echo 'for: 2m stops one-sample spikes from paging'
You will see:
for: 2m stops one-sample spikes from paging
What it means: For.
3. Build the main pieces
Step 4: Alertmanager config
Why: Routes. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: am. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 10: d1
Type this:
printf '%s\n' 'route:' ' receiver: oncall' ' group_by: [alertname]' ' group_wait: 30s' 'receivers:' '- name: oncall' ' webhook_configs:' ' - url: http://webhook:8080/alert' > /tmp/am92/alertmanager.yml && cat /tmp/am92/alertmanager.yml
You will see:
- url: http://webhook:8080/alert
What it means: AM.
Example 11: d2
Type this:
echo 'docker run -d -p 9093:9093 -v /tmp/am92/alertmanager.yml:/etc/alertmanager/alertmanager.yml prom/alertmanager'
You will see:
docker run -d -p 9093:9093 -v /tmp/am92/alertmanager.yml:/etc/alertmanager/alertmanager.yml prom/alertmanager
What it means: Run.
Example 12: d3
Type this:
echo 'Point Prometheus alerting.alertmanagers at alertmanager:9093'
You will see:
Point Prometheus alerting.alertmanagers at alertmanager:9093
What it means: Wire.
Step 5: Webhook receiver
Why: See the POST. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: hook. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 13: e1
Type this:
printf '%s\n' 'Use a tiny HTTP server that prints JSON bodies' 'Or a Slack incoming webhook URL in production' > /tmp/am92/HOOK.txt && cat /tmp/am92/HOOK.txt
You will see:
Or a Slack incoming webhook URL in production
What it means: Hook.
Example 14: e2
Type this:
echo 'curl the Alertmanager API /api/v2/alerts to list firing'
You will see:
curl the Alertmanager API /api/v2/alerts to list firing
What it means: API.
Example 15: e3
Type this:
printf '%s\n' 'Acceptance: forced HighErrorRate appears on the webhook within 3 minutes' > /tmp/am92/ACCEPT.txt && cat /tmp/am92/ACCEPT.txt
You will see:
Acceptance: forced HighErrorRate appears on the webhook within 3 minutes
What it means: Gate.
Step 6: Silence
Why: Temporary mute. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: silence. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 16: f1
Type this:
echo 'amtool silence add alertname=HighErrorRate --duration=1h --comment=maintenance'
You will see:
amtool silence add alertname=HighErrorRate --duration=1h --comment=maintenance
What it means: Silence.
Example 17: f2
Type this:
echo 'Silences must expire; never mute forever'
You will see:
Silences must expire; never mute forever
What it means: Expire.
Example 18: f3
Type this:
echo 'Document the change ticket in the silence comment'
You will see:
Document the change ticket in the silence comment
What it means: Ticket.
4. Automate and verify
Step 7: Severity routing
Why: page vs ticket. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: sev. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 19: g1
Type this:
printf '%s\n' 'severity=page -> webhook/phone' 'severity=ticket -> email/Jira' > /tmp/am92/SEV.txt && cat /tmp/am92/SEV.txt
You will see:
severity=page -> webhook/phone
What it means: Sev.
Example 20: g2
Type this:
echo 'Match on labels in Alertmanager routes'
You will see:
Match on labels in Alertmanager routes
What it means: Match.
Example 21: g3
Type this:
echo 'Inhibit rules suppress child alerts when parent fires'
You will see:
Inhibit rules suppress child alerts when parent fires
What it means: Inhibit.
Step 8: Test drill
Why: Prove paging. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: drill. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 22: h1
Type this:
echo 'Force the expr true with a fake high error metric'
You will see:
Force the expr true with a fake high error metric
What it means: Force.
Example 23: h2
Type this:
echo 'Confirm webhook payload includes summary annotation'
You will see:
Confirm webhook payload includes summary annotation
What it means: Payload.
Example 24: h3
Type this:
echo 'Page a real human in staging once a quarter'
You will see:
Page a real human in staging once a quarter
What it means: Human.
Step 9: Runbooks
Why: Link from alert. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: runbook. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 25: i1
Type this:
printf '%s\n' 'annotations.runbook_url points to a short fix page' 'On-call follows the runbook before improvising' > /tmp/am92/RUNBOOK.txt && cat /tmp/am92/RUNBOOK.txt
You will see:
annotations.runbook_url points to a short fix page
What it means: Runbook.
Example 26: i2
Type this:
echo 'Keep runbooks in git next to the rules'
You will see:
Keep runbooks in git next to the rules
What it means: Git.
Example 27: i3
Type this:
echo 'Update the runbook after every messy page'
You will see:
Update the runbook after every messy page
What it means: Update.
5. Harden and accept
Step 10: Story Paytm
Why: Noida lesson. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 28: j1
Type this:
printf '%s\n' 'At Paytm in Noida, disk filled overnight and paging hit a dead alias.' 'Farhan routed Alertmanager to a live on-call webhook.' > /tmp/am92/STORY.txt && wc -l /tmp/am92/STORY.txt
You will see:
2 /tmp/am92/STORY.txt
What it means: Story.
Example 29: j2
Type this:
echo 'Rough cost: about Rs 18 lakh in morning incident overtime.'
You will see:
Rough cost: about Rs 18 lakh in morning incident overtime.
What it means: Cost.
Example 30: j3
Type this:
echo 'Farhan: PASS'
You will see:
Farhan: PASS
What it means: Sign-off.
Step 11: Proof
Why: Files. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 31: k1
Type this:
ls /tmp/am92
You will see:
NOTES.txt\nSTORY.txt\nACCEPT.txt
What it means: Tree.
Example 32: k2
Type this:
test -f /tmp/am92/ACCEPT.txt && echo ACCEPT_OK
You will see:
ACCEPT_OK
What it means: Accept present.
Example 33: k3
Type this:
echo 'Next: ELK/EFK logging project 93'
You will see:
Next: ELK/EFK logging project 93
What it means: Bridge.
Step 12: Acceptance
Why: Webhook alert path works. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.
What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.
Example 34: z1
Type this:
echo 'Rule fires'
You will see:
Rule fires
What it means: A.
Example 35: z2
Type this:
echo 'Webhook received'
You will see:
Webhook received
What it means: B.
Example 36: z3
Type this:
echo 'Silence mutes'
You will see:
Silence mutes
What it means: C.
Check yourself
Q1. Who fixed Paytm paging?
Farhan
Q2. Router?
Alertmanager
Q3. Mute?
silence
Q4. Wait field?
for
Q5. Dead alias OK?
no
Q6. Test path?
yes
Friends, you routed a real alert. Did you understand? Let us try it once more. Fire, silence, then expire the silence and prove it pages again.
मित्रांनो, तुम्ही real alert route केला. Fire, silence, मग silence संपून पुन्हा page होते हे सिद्ध करा.
दोस्तों, आपने real alert route किया. Fire, silence, फिर silence खत्म कर फिर page हो, यह साबित करें.
Lab: try at home
- Alerting rule.
- Alertmanager.
- Webhook fires.
- Silence works.
Recap
- Alert on symptoms.
- Use for duration.
- Route by severity.
- Silences expire.
- Test the path.