Guides
How to Send Alerts from Alertmanager to Slack
By Ravindra Bagale, DevOps trainer · LinkedIn · Published: · Last updated:
Write a Prometheus rule, route it through Alertmanager to a Slack webhook, test-fire, group noisy alerts, and store the webhook as a secret.
Friends, alerts without a route are just red text. Chala mitrano — group, severity-route, and silence with a comment.
मित्रांनो, route नसलेल्या alerts फक्त लाल मजकूर. Chala mitrano — group, severity, silence with comment.
मित्रों, बिना route alerts सिर्फ लाल टेक्स्ट। Chala mitrano — group, severity, silence with comment।
Quick answer
Hold this vertical picture:
- Write rules.yml.
- Configure Slack receiver.
- Point Prometheus at Alertmanager.
- Manual test fire.
- Group, severity-route, silence with comment.
Language: English · मराठी · हिंदी
Alert to Slack
Why Alertmanager
Prometheus detects; Alertmanager notifies and quiets.
Why: Direct email from every replica floods inboxes.
What: route + receivers + group.
How: Slack is a common first channel.
Example 1: Simple alert rule
Type this:
mkdir -p /tmp/am && printf '%s\n' 'groups:' '- name: demo' ' rules:' ' - alert: InstanceDown' ' expr: up == 0' ' for: 1m' ' labels: {severity: page}' ' annotations: {summary: "Target down"}' > /tmp/am/rules.yml && cat /tmp/am/rules.yml
You will see:
What it means: for: 1m stops flapping one-second blips from paging you.
Example 2: Prometheus alertmanager target
Type this:
printf '%s\n' 'alerting:' ' alertmanagers:' ' - static_configs:' ' - targets: ["alertmanager:9093"]'
You will see:
What it means: Prometheus sends firing alerts to Alertmanager.
Example 3: Alertmanager Slack receiver
Type this:
printf '%s\n' 'route:' ' receiver: slack-ops' 'receivers:' '- name: slack-ops' ' slack_configs:' ' - api_url: "https://hooks.slack.example.internal/services/DEMO/DEMO/DEMO"' ' channel: "#ops-alerts"' ' send_resolved: true' > /tmp/am/alertmanager.yml && grep channel /tmp/am/alertmanager.yml
You will see:
What it means: Use a real Incoming Webhook URL from Slack admin — demo host only here.
Concept 1: Fire and prove
Rules + test API.
Why: Untested alerts fail on the first outage.
What: for: duration; curl API v2 alerts.
How: send_resolved true.
Example 4: Test alert with curl API
Type this:
printf '%s\n' 'curl -H "Content-Type: application/json" -d "[{\"labels\":{\"alertname\":\"ManualTest\",\"severity\":\"page\"}}]" http://127.0.0.1:9093/api/v2/alerts'
You will see:
curl -H "Content-Type: application/json" -d "[{"labels":{"alertname":"ManualTest","severity":"page"}}]" http://127.0.0.1:9093/api/v2/alerts
What it means: Manual fire proves Slack without waiting for a real outage.
Example 7: Resolved notifications
Type this:
printf '%s\n' 'send_resolved: true # tell Slack when it cleared'
You will see:
send_resolved: true # tell Slack when it cleared
What it means: On-call sleeps better when green also arrives.
Example 10: Check Alertmanager UI
Type this:
printf '%s\n' 'Open :9093 → Alerts / Silences to see firing and muted'
You will see:
Open :9093 → Alerts / Silences to see firing and muted
What it means: UI confirms what Slack should show.
Concept 2: Noise control
group_by, wait, silence.
Why: Fifty messages for one blip burn trust.
What: group_wait; amtool silence.
How: repeat_interval.
Example 5: Inhibit / group idea
Type this:
printf '%s\n' 'group_by: [alertname, severity]' 'group_wait: 30s'
You will see:
group_by: [alertname, severity]
group_wait: 30s
What it means: Grouping prevents 50 Slack messages for one incident.
Example 6: Silence sketch
Type this:
printf '%s\n' 'amtool silence add alertname=ManualTest --duration=1h --comment="maint"'
You will see:
amtool silence add alertname=ManualTest --duration=1h --comment="maint"
What it means: Silence during maintenance — say why.
Example 11: Repeat interval
Type this:
printf '%s\n' 'repeat_interval: 4h'
You will see:
What it means: Still-firing alerts re-notify on this cadence.
Concept 3: Route and secrets
severity routes; protect webhook.
Why: Leaked webhooks spam customers.
What: match severity; secret store.
How: Runbook annotation links.
Example 8: Severity routing
Type this:
printf '%s\n' 'routes:' '- match: {severity: page}' ' receiver: slack-ops' '- match: {severity: ticket}' ' receiver: slack-low'
You will see:
What it means: Not every alert should page the phone.
Example 9: Webhook secret handling
Type this:
printf '%s\n' 'ok: store Slack webhook in a secret manager or sealed file — not plain git'
You will see:
ok: store Slack webhook in a secret manager or sealed file — not plain git
What it means: Leaked webhooks spam channels — Capgemini finding.
Example 12: Runbook link in annotation
Type this:
printf '%s\n' 'annotations: {runbook: "https://wiki.example.internal/runbooks/instance-down"}'
You will see:
annotations: {runbook: "https://wiki.example.internal/runbooks/instance-down"}
What it means: Slack message should deep-link the fix steps.
Alertmanager card
1 rules.yml 2 Alertmanager receiver Slack 3 wire Prometheus alerting 4 manual test fire 5 group/silence 6 secret webhook
Suppose: No Slack message
Check route receiver name, webhook URL, and Alertmanager logs.
Suppose: Alert storm
Tighten group_by and for:; add silences for maint.
Ravindra Bagale's Tip
At Amazon-style on-call drills, a manual test alert to Slack is required before the service goes live. Reviewers ask where the webhook is stored.
Amazon-style drills मध्ये Slack वर manual test alert आधी. Reviewers webhook कुठे साठवला विचारतात.
Amazon-style drills में Slack पर manual test alert पहले। Reviewers webhook कहाँ रखा पूछते हैं।
Check yourself (quiz)
Q1. Who pages Slack?
Alertmanager receiver
Q2. Why for: 1m?
Avoid flap pages
Q3. Test without outage?
POST to Alertmanager API
Q4. Silence when?
Maintenance windows
Q5. Webhook in git?
No — secret store
Q6. group_by?
Collapse related alerts
Reading tracking
Mark this guide when you finish. Your browser remembers it on this device.
rule → Alertmanager → Slack → prove → quiet noise.
Did you understand? Let us try it once more.
Frequently asked questions
Prometheus vs Alertmanager?
Prom evaluates; AM routes/notifies.
Why send_resolved?
Know when it cleared.
PagerDuty?
Same receivers idea — different config block.
Alert spam?
group_by, for, inhibit, silence.
Secure webhook?
Secret manager / sealed secret.
Runbooks?
Put URL in annotations.