Ravindra BagaleCourses & study guides मराठी Track your progress

Guides

How to Send Alerts from Alertmanager to Slack

Write a Prometheus rule, route it through Alertmanager to a Slack webhook, test-fire, group noisy alerts, and store the webhook as a secret.

Friends, alerts without a route are just red text. Chala mitrano — group, severity-route, and silence with a comment.

Quick answer

Hold this vertical picture:

  1. Write rules.yml.
  2. Configure Slack receiver.
  3. Point Prometheus at Alertmanager.
  4. Manual test fire.
  5. Group, severity-route, silence with comment.

Language: English · मराठी · हिंदी

Prometheus ruleexpr + for Alertmanagerroute + group Slack webhookChannel post On-call sees itAct or silence ResolvedGreen follow-up
Alert to Slack

Why Alertmanager

Prometheus detects; Alertmanager notifies and quiets.

Why: Direct email from every replica floods inboxes. What: route + receivers + group. How: Slack is a common first channel.

Example 1: Simple alert rule

Type this:

mkdir -p /tmp/am && printf '%s\n' 'groups:' '- name: demo' '  rules:' '  - alert: InstanceDown' '    expr: up == 0' '    for: 1m' '    labels: {severity: page}' '    annotations: {summary: "Target down"}' > /tmp/am/rules.yml && cat /tmp/am/rules.yml

You will see:

groups:

What it means: for: 1m stops flapping one-second blips from paging you.

Example 2: Prometheus alertmanager target

Type this:

printf '%s\n' 'alerting:' '  alertmanagers:' '  - static_configs:' '    - targets: ["alertmanager:9093"]'

You will see:

alerting:

What it means: Prometheus sends firing alerts to Alertmanager.

Example 3: Alertmanager Slack receiver

Type this:

printf '%s\n' 'route:' '  receiver: slack-ops' 'receivers:' '- name: slack-ops' '  slack_configs:' '  - api_url: "https://hooks.slack.example.internal/services/DEMO/DEMO/DEMO"' '    channel: "#ops-alerts"' '    send_resolved: true' > /tmp/am/alertmanager.yml && grep channel /tmp/am/alertmanager.yml

You will see:

channel: "#ops-alerts"

What it means: Use a real Incoming Webhook URL from Slack admin — demo host only here.

Concept 1: Fire and prove

Rules + test API.

Why: Untested alerts fail on the first outage. What: for: duration; curl API v2 alerts. How: send_resolved true.

Example 4: Test alert with curl API

Type this:

printf '%s\n' 'curl -H "Content-Type: application/json" -d "[{\"labels\":{\"alertname\":\"ManualTest\",\"severity\":\"page\"}}]" http://127.0.0.1:9093/api/v2/alerts'

You will see:

curl -H "Content-Type: application/json" -d "[{"labels":{"alertname":"ManualTest","severity":"page"}}]" http://127.0.0.1:9093/api/v2/alerts

What it means: Manual fire proves Slack without waiting for a real outage.

Example 7: Resolved notifications

Type this:

printf '%s\n' 'send_resolved: true  # tell Slack when it cleared'

You will see:

send_resolved: true  # tell Slack when it cleared

What it means: On-call sleeps better when green also arrives.

Example 10: Check Alertmanager UI

Type this:

printf '%s\n' 'Open :9093 → Alerts / Silences to see firing and muted'

You will see:

Open :9093 → Alerts / Silences to see firing and muted

What it means: UI confirms what Slack should show.

Concept 2: Noise control

group_by, wait, silence.

Why: Fifty messages for one blip burn trust. What: group_wait; amtool silence. How: repeat_interval.

Example 5: Inhibit / group idea

Type this:

printf '%s\n' 'group_by: [alertname, severity]' 'group_wait: 30s'

You will see:

group_by: [alertname, severity]
group_wait: 30s

What it means: Grouping prevents 50 Slack messages for one incident.

Example 6: Silence sketch

Type this:

printf '%s\n' 'amtool silence add alertname=ManualTest --duration=1h --comment="maint"'

You will see:

amtool silence add alertname=ManualTest --duration=1h --comment="maint"

What it means: Silence during maintenance — say why.

Example 11: Repeat interval

Type this:

printf '%s\n' 'repeat_interval: 4h'

You will see:

repeat_interval: 4h

What it means: Still-firing alerts re-notify on this cadence.

Concept 3: Route and secrets

severity routes; protect webhook.

Why: Leaked webhooks spam customers. What: match severity; secret store. How: Runbook annotation links.

Example 8: Severity routing

Type this:

printf '%s\n' 'routes:' '- match: {severity: page}' '  receiver: slack-ops' '- match: {severity: ticket}' '  receiver: slack-low'

You will see:

routes:

What it means: Not every alert should page the phone.

Example 9: Webhook secret handling

Type this:

printf '%s\n' 'ok: store Slack webhook in a secret manager or sealed file — not plain git'

You will see:

ok: store Slack webhook in a secret manager or sealed file — not plain git

What it means: Leaked webhooks spam channels — Capgemini finding.

Example 12: Runbook link in annotation

Type this:

printf '%s\n' 'annotations: {runbook: "https://wiki.example.internal/runbooks/instance-down"}'

You will see:

annotations: {runbook: "https://wiki.example.internal/runbooks/instance-down"}

What it means: Slack message should deep-link the fix steps.

Alertmanager card

1 rules.yml 2 Alertmanager receiver Slack 3 wire Prometheus alerting 4 manual test fire 5 group/silence 6 secret webhook

Suppose: No Slack message

Check route receiver name, webhook URL, and Alertmanager logs.

Suppose: Alert storm

Tighten group_by and for:; add silences for maint.

Ravindra Bagale's Tip

At Amazon-style on-call drills, a manual test alert to Slack is required before the service goes live. Reviewers ask where the webhook is stored.

Check yourself (quiz)

Q1. Who pages Slack?

Alertmanager receiver

Q2. Why for: 1m?

Avoid flap pages

Q3. Test without outage?

POST to Alertmanager API

Q4. Silence when?

Maintenance windows

Q5. Webhook in git?

No — secret store

Q6. group_by?

Collapse related alerts

Reading tracking

Mark this guide when you finish. Your browser remembers it on this device.

rule → Alertmanager → Slack → prove → quiet noise. Did you understand? Let us try it once more.

Frequently asked questions

Prometheus vs Alertmanager?

Prom evaluates; AM routes/notifies.

Why send_resolved?

Know when it cleared.

PagerDuty?

Same receivers idea — different config block.

Alert spam?

group_by, for, inhibit, silence.

Secure webhook?

Secret manager / sealed secret.

Runbooks?

Put URL in annotations.