Chapter 83: DevOps Projects Part 13: Monitoring Prometheus Grafana Alertmanager EFK Loki CloudWatch OpenTelemetry

83.1 Project 91: Prometheus + Grafana dashboard

By Ravindra Bagale Last updated: 28 October 2026

Friends, guessing is not monitoring. Today we scrape a /metrics endpoint with Prometheus and chart it in Grafana.

What you will build in this project

Run Prometheus.
Scrape an app metric.
Run Grafana.
Build a dashboard panel.

1. Start from a real company story

At Dream11 in Mumbai, CPU saturation was invisible until users churned during a match.

Rough cost: about Rs 40 lakh in lost fantasy entries that hour.

Rohit put RED metrics on Grafana with Prometheus scrape.

Today we chart one metric end to end.

Picture 1: app /metrics -> Prometheus scrape -> Grafana panel

Ravindra Bagale's Tip

Students scrape every 1s and store forever. Start at 15s scrape and set retention. Keep that habit.

2. Start and prepare

Step 1: Why metrics

Why: Guessing fails. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 1: a1

Type this:

echo 'Numbers beat opinions during an incident'

You will see:

Numbers beat opinions during an incident

What it means: Why.

Example 2: a2

Type this:

mkdir -p /tmp/prom91 && echo 'Today: Prometheus, scrape, Grafana, panel'

You will see:

Today: Prometheus, scrape, Grafana, panel

What it means: Scope.

Example 3: a3

Type this:

echo 'RED: rate, errors, duration for request services'

You will see:

RED: rate, errors, duration for request services

What it means: RED.

Step 2: Prometheus config

Why: Scrape job. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: config. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 4: b1

Type this:

printf '%s\n' 'global:' '  scrape_interval: 15s' 'scrape_configs:' '- job_name: shop-api' '  static_configs:' '  - targets: ["app:8080"]' > /tmp/prom91/prometheus.yml && cat /tmp/prom91/prometheus.yml

You will see:

scrape_interval: 15s

What it means: Config.

Example 5: b2

Type this:

echo 'docker run -d -p 9090:9090 -v /tmp/prom91/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus'

You will see:

docker run -d -p 9090:9090 -v /tmp/prom91/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

What it means: Run.

Example 6: b3

Type this:

echo 'Open http://localhost:9090/targets and wait for UP'

You will see:

Open http://localhost:9090/targets and wait for UP

What it means: Targets.

Step 3: App metrics

Why: Expose /metrics. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: app. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 7: c1

Type this:

printf '%s\n' 'Expose Prometheus text format on /metrics' 'Include http_requests_total and duration histogram' > /tmp/prom91/APP.txt && cat /tmp/prom91/APP.txt

You will see:

Include http_requests_total and duration histogram

What it means: App.

Example 8: c2

Type this:

echo 'curl -s http://app:8080/metrics | head'

You will see:

curl -s http://app:8080/metrics | head

What it means: Curl.

Example 9: c3

Type this:

echo 'Client libraries exist for Go, Java, Python, Node'

You will see:

Client libraries exist for Go, Java, Python, Node

What it means: Libs.

3. Build the main pieces

Step 4: PromQL

Why: Query. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: query. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 10: d1

Type this:

echo 'rate(http_requests_total[1m])'

You will see:

rate(http_requests_total[1m])

What it means: Rate.

Example 11: d2

Type this:

echo 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))'

You will see:

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

What it means: p95.

Example 12: d3

Type this:

echo 'Try queries in the Prometheus Graph UI first'

You will see:

Try queries in the Prometheus Graph UI first

What it means: UI.

Step 5: Grafana

Why: Datasource. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: grafana. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 13: e1

Type this:

echo 'docker run -d -p 3000:3000 grafana/grafana'

You will see:

docker run -d -p 3000:3000 grafana/grafana

What it means: Run.

Example 14: e2

Type this:

printf '%s\n' 'Add datasource Prometheus URL http://prometheus:9090' 'Create dashboard then Add panel with PromQL rate query' > /tmp/prom91/GRAFANA.txt && cat /tmp/prom91/GRAFANA.txt

You will see:

Create dashboard then Add panel with PromQL rate query

What it means: Panel.

Example 15: e3

Type this:

printf '%s\n' 'Acceptance: panel shows request rate moving when you hit the app' > /tmp/prom91/ACCEPT.txt && cat /tmp/prom91/ACCEPT.txt

You will see:

Acceptance: panel shows request rate moving when you hit the app

What it means: Gate.

Step 6: Dashboard tips

Why: Readable. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: tips. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 16: f1

Type this:

printf '%s\n' 'One row: traffic' 'One row: errors' 'One row: latency' 'One row: saturation' > /tmp/prom91/DASH.txt && cat /tmp/prom91/DASH.txt

You will see:

One row: latency

What it means: Dash.

Example 17: f2

Type this:

echo 'Use job and instance template variables'

You will see:

Use job and instance template variables

What it means: Vars.

Example 18: f3

Type this:

echo 'Save JSON to git so dashboards are code'

You will see:

Save JSON to git so dashboards are code

What it means: As code.

4. Automate and verify

Step 7: Retention

Why: Disk. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: retain. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 19: g1

Type this:

echo 'Set --storage.tsdb.retention.time=15d on Prometheus'

You will see:

Set --storage.tsdb.retention.time=15d on Prometheus

What it means: Retain.

Example 20: g2

Type this:

echo 'Remote write to long-term storage when you outgrow local disk'

You will see:

Remote write to long-term storage when you outgrow local disk

What it means: Remote.

Example 21: g3

Type this:

echo '15s scrape times many series means plan disk early'

You will see:

15s scrape times many series means plan disk early

What it means: Math.

Step 8: Labels

Why: Cardinality. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: labels. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 22: h1

Type this:

echo 'Never put user ids or emails in metric labels'

You will see:

Never put user ids or emails in metric labels

What it means: Rule.

Example 23: h2

Type this:

echo 'Keep label sets bounded: method, status, route group'

You will see:

Keep label sets bounded: method, status, route group

What it means: Bound.

Example 24: h3

Type this:

echo 'High cardinality kills Prometheus memory'

You will see:

High cardinality kills Prometheus memory

What it means: Risk.

Step 9: Recording rules

Why: Precompute. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: rules. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 25: i1

Type this:

printf '%s\n' 'groups:' '- name: shop' '  rules:' '  - record: job:http_requests:rate1m' '    expr: sum(rate(http_requests_total[1m]))' > /tmp/prom91/RULES.txt && cat /tmp/prom91/RULES.txt

You will see:

expr: sum(rate(http_requests_total[1m]))

What it means: Rules.

Example 26: i2

Type this:

echo 'Dashboards query recording rules for speed'

You will see:

Dashboards query recording rules for speed

What it means: Speed.

Example 27: i3

Type this:

echo 'Name recording rules with a clear prefix'

You will see:

Name recording rules with a clear prefix

What it means: Name.

5. Harden and accept

Step 10: Story Dream11

Why: Blind during match. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 28: j1

Type this:

printf '%s\n' 'At Dream11 in Mumbai, CPU saturation was invisible during a match.' 'Rohit put RED metrics on Grafana with Prometheus.' > /tmp/prom91/STORY.txt && wc -l /tmp/prom91/STORY.txt

You will see:

2 /tmp/prom91/STORY.txt

What it means: Story.

Example 29: j2

Type this:

echo 'Rough cost: about Rs 40 lakh in lost fantasy entries that hour'

You will see:

Rough cost: about Rs 40 lakh in lost fantasy entries that hour

What it means: Cost.

Example 30: j3

Type this:

echo 'Rohit: PASS'

You will see:

Rohit: PASS

What it means: Sign-off.

Step 11: Proof

Why: Files. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 31: k1

Type this:

ls /tmp/prom91

You will see:

ACCEPT.txt\nAPP.txt\nDASH.txt\nGRAFANA.txt\nRULES.txt\nSTORY.txt\nprometheus.yml

What it means: Tree.

Example 32: k2

Type this:

grep scrape_interval /tmp/prom91/prometheus.yml

You will see:

scrape_interval: 15s

What it means: Interval.

Example 33: k3

Type this:

echo 'Next: Alertmanager project 92'

You will see:

Next: Alertmanager project 92

What it means: Bridge.

Step 12: Acceptance

Why: Live panel moves. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 34: z1

Type this:

echo 'Target UP in Prometheus'

You will see:

Target UP in Prometheus

What it means: Target.

Example 35: z2

Type this:

echo 'Grafana panel charts rate'

You will see:

Grafana panel charts rate

What it means: Panel.

Example 36: z3

Type this:

echo 'Stale target is visible'

You will see:

Stale target is visible

What it means: Stale.

Check yourself

Q1. Who fixed Dream11 blind spot?

Rohit

Q2. Scrape tool?

Prometheus

Q3. UI?

Grafana

Q4. Path?

/metrics

Q5. Scrape default tip?

15s

Q6. Forever retention?

Friends, you charted a live metric. Did you understand? Let us try it once more. Break the scrape target and prove the panel goes stale, then fix it.

Lab: try at home

Prometheus up.
Target scraped.
Grafana dashboard.
Stale then fixed.

Recap

Scrape not push first.
RED or USE metrics.
15s is enough.
Retention matters.
Dashboard tells a story.