Ravindra BagaleCourses & study guides मराठी Track your progress

Guides

Alertmanager से Slack पर Alerts कैसे भेजें

Prometheus rule → Alertmanager → Slack webhook → test fire → group/silence → secret webhook.

Friends, alerts without a route are just red text. Chala mitrano — group, severity-route, and silence with a comment.

Quick answer

Hold this vertical picture:

  1. rules.yml.
  2. Slack receiver.
  3. Prom → AM.
  4. test fire.
  5. group; severity; silence.

Language: English · मराठी · हिंदी

Prometheus ruleexpr + for Alertmanagerroute + group Slack webhookChannel post On-call sees itAct or silence ResolvedGreen follow-up
Alert to Slack

Alertmanager क्यों

detect vs notify.

Why: direct flood. What: route+group. How: Slack first.

Example 1: Simple alert rule

Type this:

mkdir -p /tmp/am && printf '%s\n' 'groups:' '- name: demo' '  rules:' '  - alert: InstanceDown' '    expr: up == 0' '    for: 1m' '    labels: {severity: page}' '    annotations: {summary: "Target down"}' > /tmp/am/rules.yml && cat /tmp/am/rules.yml

You will see:

groups:

What it means: for: 1m stops flapping one-second blips from paging you.

Example 2: Prometheus alertmanager target

Type this:

printf '%s\n' 'alerting:' '  alertmanagers:' '  - static_configs:' '    - targets: ["alertmanager:9093"]'

You will see:

alerting:

What it means: Prometheus sends firing alerts to Alertmanager.

Example 3: Alertmanager Slack receiver

Type this:

printf '%s\n' 'route:' '  receiver: slack-ops' 'receivers:' '- name: slack-ops' '  slack_configs:' '  - api_url: "https://hooks.slack.example.internal/services/DEMO/DEMO/DEMO"' '    channel: "#ops-alerts"' '    send_resolved: true' > /tmp/am/alertmanager.yml && grep channel /tmp/am/alertmanager.yml

You will see:

channel: "#ops-alerts"

What it means: Use a real Incoming Webhook URL from Slack admin — demo host only here.

Concept 1: Fire/prove

rules + test API.

Why: untested fail. What: for:; curl. How: send_resolved.

Example 4: Test alert with curl API

Type this:

printf '%s\n' 'curl -H "Content-Type: application/json" -d "[{\"labels\":{\"alertname\":\"ManualTest\",\"severity\":\"page\"}}]" http://127.0.0.1:9093/api/v2/alerts'

You will see:

curl -H "Content-Type: application/json" -d "[{"labels":{"alertname":"ManualTest","severity":"page"}}]" http://127.0.0.1:9093/api/v2/alerts

What it means: Manual fire proves Slack without waiting for a real outage.

Example 7: Resolved notifications

Type this:

printf '%s\n' 'send_resolved: true  # tell Slack when it cleared'

You will see:

send_resolved: true  # tell Slack when it cleared

What it means: On-call sleeps better when green also arrives.

Example 10: Check Alertmanager UI

Type this:

printf '%s\n' 'Open :9093 → Alerts / Silences to see firing and muted'

You will see:

Open :9093 → Alerts / Silences to see firing and muted

What it means: UI confirms what Slack should show.

Concept 2: Noise

group/silence.

Why: 50 messages. What: group_wait; silence. How: repeat_interval.

Example 5: Inhibit / group idea

Type this:

printf '%s\n' 'group_by: [alertname, severity]' 'group_wait: 30s'

You will see:

group_by: [alertname, severity]
group_wait: 30s

What it means: Grouping prevents 50 Slack messages for one incident.

Example 6: Silence sketch

Type this:

printf '%s\n' 'amtool silence add alertname=ManualTest --duration=1h --comment="maint"'

You will see:

amtool silence add alertname=ManualTest --duration=1h --comment="maint"

What it means: Silence during maintenance — say why.

Example 11: Repeat interval

Type this:

printf '%s\n' 'repeat_interval: 4h'

You will see:

repeat_interval: 4h

What it means: Still-firing alerts re-notify on this cadence.

Concept 3: Route/secrets

severity; protect URL.

Why: leaked spam. What: match; secret. How: runbook link.

Example 8: Severity routing

Type this:

printf '%s\n' 'routes:' '- match: {severity: page}' '  receiver: slack-ops' '- match: {severity: ticket}' '  receiver: slack-low'

You will see:

routes:

What it means: Not every alert should page the phone.

Example 9: Webhook secret handling

Type this:

printf '%s\n' 'ok: store Slack webhook in a secret manager or sealed file — not plain git'

You will see:

ok: store Slack webhook in a secret manager or sealed file — not plain git

What it means: Leaked webhooks spam channels — Capgemini finding.

Example 12: Runbook link in annotation

Type this:

printf '%s\n' 'annotations: {runbook: "https://wiki.example.internal/runbooks/instance-down"}'

You will see:

annotations: {runbook: "https://wiki.example.internal/runbooks/instance-down"}

What it means: Slack message should deep-link the fix steps.

Alertmanager card

1 rules 2 Slack 3 wire 4 test 5 group/silence 6 secret

Suppose: no Slack

route/URL/logs.

Suppose: storm

group + for.

Ravindra Bagale's Tip

At Amazon-style on-call drills, a manual test alert to Slack is required before the service goes live. Reviewers ask where the webhook is stored.

Check yourself (quiz)

Q1. Slack?

AM receiver

Q2. for:1m?

no flap

Q3. test?

API POST

Q4. silence?

maint

Q5. webhook git?

no

Q6. group_by?

collapse

Reading tracking

Mark this guide when you finish. Your browser remembers it on this device.

rule → AM → Slack → prove → quiet. Did you understand? Let us try it once more.

Frequently asked questions

Prometheus vs Alertmanager?

Prom evaluates; AM routes/notifies.

Why send_resolved?

Know when it cleared.

PagerDuty?

Same receivers idea — different config block.

Alert spam?

group_by, for, inhibit, silence.

Secure webhook?

Secret manager / sealed secret.

Runbooks?

Put URL in annotations.