Guides
Alertmanager से Slack पर Alerts कैसे भेजें
By Ravindra Bagale, DevOps trainer · LinkedIn · Published: · Last updated:
Prometheus rule → Alertmanager → Slack webhook → test fire → group/silence → secret webhook.
Friends, alerts without a route are just red text. Chala mitrano — group, severity-route, and silence with a comment.
मित्रांनो, route नसलेल्या alerts फक्त लाल मजकूर. Chala mitrano — group, severity, silence with comment.
मित्रों, बिना route alerts सिर्फ लाल टेक्स्ट। Chala mitrano — group, severity, silence with comment।
Quick answer
Hold this vertical picture:
- rules.yml.
- Slack receiver.
- Prom → AM.
- test fire.
- group; severity; silence.
Language: English · मराठी · हिंदी
Alert to Slack
Alertmanager क्यों
detect vs notify.
Why: direct flood.
What: route+group.
How: Slack first.
Example 1: Simple alert rule
Type this:
mkdir -p /tmp/am && printf '%s\n' 'groups:' '- name: demo' ' rules:' ' - alert: InstanceDown' ' expr: up == 0' ' for: 1m' ' labels: {severity: page}' ' annotations: {summary: "Target down"}' > /tmp/am/rules.yml && cat /tmp/am/rules.yml
You will see:
What it means: for: 1m stops flapping one-second blips from paging you.
Example 2: Prometheus alertmanager target
Type this:
printf '%s\n' 'alerting:' ' alertmanagers:' ' - static_configs:' ' - targets: ["alertmanager:9093"]'
You will see:
What it means: Prometheus sends firing alerts to Alertmanager.
Example 3: Alertmanager Slack receiver
Type this:
printf '%s\n' 'route:' ' receiver: slack-ops' 'receivers:' '- name: slack-ops' ' slack_configs:' ' - api_url: "https://hooks.slack.example.internal/services/DEMO/DEMO/DEMO"' ' channel: "#ops-alerts"' ' send_resolved: true' > /tmp/am/alertmanager.yml && grep channel /tmp/am/alertmanager.yml
You will see:
What it means: Use a real Incoming Webhook URL from Slack admin — demo host only here.
Concept 1: Fire/prove
rules + test API.
Why: untested fail.
What: for:; curl.
How: send_resolved.
Example 4: Test alert with curl API
Type this:
printf '%s\n' 'curl -H "Content-Type: application/json" -d "[{\"labels\":{\"alertname\":\"ManualTest\",\"severity\":\"page\"}}]" http://127.0.0.1:9093/api/v2/alerts'
You will see:
curl -H "Content-Type: application/json" -d "[{"labels":{"alertname":"ManualTest","severity":"page"}}]" http://127.0.0.1:9093/api/v2/alerts
What it means: Manual fire proves Slack without waiting for a real outage.
Example 7: Resolved notifications
Type this:
printf '%s\n' 'send_resolved: true # tell Slack when it cleared'
You will see:
send_resolved: true # tell Slack when it cleared
What it means: On-call sleeps better when green also arrives.
Example 10: Check Alertmanager UI
Type this:
printf '%s\n' 'Open :9093 → Alerts / Silences to see firing and muted'
You will see:
Open :9093 → Alerts / Silences to see firing and muted
What it means: UI confirms what Slack should show.
Concept 2: Noise
group/silence.
Why: 50 messages.
What: group_wait; silence.
How: repeat_interval.
Example 5: Inhibit / group idea
Type this:
printf '%s\n' 'group_by: [alertname, severity]' 'group_wait: 30s'
You will see:
group_by: [alertname, severity]
group_wait: 30s
What it means: Grouping prevents 50 Slack messages for one incident.
Example 6: Silence sketch
Type this:
printf '%s\n' 'amtool silence add alertname=ManualTest --duration=1h --comment="maint"'
You will see:
amtool silence add alertname=ManualTest --duration=1h --comment="maint"
What it means: Silence during maintenance — say why.
Example 11: Repeat interval
Type this:
printf '%s\n' 'repeat_interval: 4h'
You will see:
What it means: Still-firing alerts re-notify on this cadence.
Concept 3: Route/secrets
severity; protect URL.
Why: leaked spam.
What: match; secret.
How: runbook link.
Example 8: Severity routing
Type this:
printf '%s\n' 'routes:' '- match: {severity: page}' ' receiver: slack-ops' '- match: {severity: ticket}' ' receiver: slack-low'
You will see:
What it means: Not every alert should page the phone.
Example 9: Webhook secret handling
Type this:
printf '%s\n' 'ok: store Slack webhook in a secret manager or sealed file — not plain git'
You will see:
ok: store Slack webhook in a secret manager or sealed file — not plain git
What it means: Leaked webhooks spam channels — Capgemini finding.
Example 12: Runbook link in annotation
Type this:
printf '%s\n' 'annotations: {runbook: "https://wiki.example.internal/runbooks/instance-down"}'
You will see:
annotations: {runbook: "https://wiki.example.internal/runbooks/instance-down"}
What it means: Slack message should deep-link the fix steps.
Alertmanager card
1 rules 2 Slack 3 wire 4 test 5 group/silence 6 secret
Suppose: no Slack
route/URL/logs.
Suppose: storm
group + for.
Ravindra Bagale's Tip
At Amazon-style on-call drills, a manual test alert to Slack is required before the service goes live. Reviewers ask where the webhook is stored.
Amazon-style drills मध्ये Slack वर manual test alert आधी. Reviewers webhook कुठे साठवला विचारतात.
Amazon-style drills में Slack पर manual test alert पहले। Reviewers webhook कहाँ रखा पूछते हैं।
Check yourself (quiz)
Q1. Slack?
AM receiver
Q2. for:1m?
no flap
Q3. test?
API POST
Q4. silence?
maint
Q5. webhook git?
no
Q6. group_by?
collapse
Reading tracking
Mark this guide when you finish. Your browser remembers it on this device.
rule → AM → Slack → prove → quiet.
Did you understand? Let us try it once more.
Frequently asked questions
Prometheus vs Alertmanager?
Prom evaluates; AM routes/notifies.
Why send_resolved?
Know when it cleared.
PagerDuty?
Same receivers idea — different config block.
Alert spam?
group_by, for, inhibit, silence.
Secure webhook?
Secret manager / sealed secret.
Runbooks?
Put URL in annotations.