Ravindra BagaleCourses & study guides मराठी

Chapter 83: DevOps Projects Part 13: Monitoring Prometheus Grafana Alertmanager EFK Loki CloudWatch OpenTelemetry

83.5 Project 95: CloudWatch dashboards and alarms

By Last updated:

Friends, AWS has built-in metrics. Today we create a CloudWatch dashboard for an ALB and an alarm that pages when 5xx rises.

What you will build in this project

  1. Pick ALB metrics.
  2. Build a dashboard.
  3. Create a 5xx alarm.
  4. Notify SNS.

1. Start from a real company story

At MakeMyTrip in Gurgaon, an ALB 5xx spike had no alarm and support noticed from tweets.

Rough cost: about Rs 15 lakh in goodwill refunds.

Vikram put CloudWatch alarms on ALB 5xx to SNS.

Today we build that alarm path.

ALB5xxCWdashboardalarmthresholdSNSnotify
Picture 1: ALB metrics -> CloudWatch dashboard -> Alarm -> SNS

Ravindra Bagale's Tip

Students alarm on raw count without period. Use rate or Sum with a sensible period. Keep that habit.

2. Start and prepare

Step 1: Why this project

Why: Real outages teach. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 1: a1

Type this:

echo 'Today we build cw95'

You will see:

Today we build cw95

What it means: Why.

Example 2: a2

Type this:

mkdir -p /tmp/cw95 && echo 'Scope ready'

You will see:

Scope ready

What it means: Scope.

Example 3: a3

Type this:

echo 'Keep commands short and check every output'

You will see:

Keep commands short and check every output

What it means: Habit.

Step 2: Prepare workspace

Why: Folders and notes. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: prep. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 4: b1

Type this:

mkdir -p /tmp/cw95 && printf '%s\n' 'lab notes' > /tmp/cw95/NOTES.txt && cat /tmp/cw95/NOTES.txt

You will see:

lab notes

What it means: Notes.

Example 5: b2

Type this:

echo 'Work only under /tmp/cw95 for this project'

You will see:

Work only under /tmp/cw95 for this project

What it means: Path.

Example 6: b3

Type this:

echo 'Write ACCEPT.txt at the end'

You will see:

Write ACCEPT.txt at the end

What it means: Accept file.

Step 3: ALB metrics

Why: What to watch. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: metrics. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 7: c1

Type this:

printf '%s\n' 'HTTPCode_Target_5XX_Count' 'TargetResponseTime' 'HealthyHostCount' 'RequestCount' > /tmp/cw95/METRICS.txt && cat /tmp/cw95/METRICS.txt

You will see:

HTTPCode_Target_5XX_Count

What it means: Metrics.

Example 8: c2

Type this:

echo 'Namespace AWS/ApplicationELB with LoadBalancer dimension'

You will see:

Namespace AWS/ApplicationELB with LoadBalancer dimension

What it means: NS.

Example 9: c3

Type this:

echo 'Prefer target 5xx over ELB 5xx for app bugs'

You will see:

Prefer target 5xx over ELB 5xx for app bugs

What it means: Prefer.

3. Build the main pieces

Step 4: Dashboard

Why: Widgets. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: dash. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 10: d1

Type this:

printf '%s\n' 'aws cloudwatch put-dashboard --dashboard-name shop-alb --dashboard-body file://dash.json' > /tmp/cw95/DASH.txt && cat /tmp/cw95/DASH.txt

You will see:

aws cloudwatch put-dashboard --dashboard-name shop-alb --dashboard-body file://dash.json

What it means: Dash.

Example 11: d2

Type this:

echo 'One row traffic, one row errors, one row latency'

You will see:

One row traffic, one row errors, one row latency

What it means: Layout.

Example 12: d3

Type this:

echo 'Store dashboard JSON in git'

You will see:

Store dashboard JSON in git

What it means: Git.

Step 5: Alarm

Why: Threshold. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: alarm. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 13: e1

Type this:

printf '%s\n' 'aws cloudwatch put-metric-alarm --alarm-name shop-alb-5xx' '  --metric-name HTTPCode_Target_5XX_Count --namespace AWS/ApplicationELB' '  --statistic Sum --period 60 --threshold 20 --comparison-operator GreaterThanThreshold' '  --evaluation-periods 3 --alarm-actions arn:aws:sns:ap-south-1:123:oncall' > /tmp/cw95/ALARM.txt && cat /tmp/cw95/ALARM.txt

You will see:

--statistic Sum --period 60 --threshold 20 --comparison-operator GreaterThanThreshold

What it means: Alarm.

Example 14: e2

Type this:

echo 'evaluation-periods 3 reduces flappy pages'

You will see:

evaluation-periods 3 reduces flappy pages

What it means: Eval.

Example 15: e3

Type this:

printf '%s\n' 'Acceptance: set_alarm_state ALARM publishes to SNS' > /tmp/cw95/ACCEPT.txt && cat /tmp/cw95/ACCEPT.txt

You will see:

Acceptance: set_alarm_state ALARM publishes to SNS

What it means: Gate.

Step 6: SNS topic

Why: Notify humans. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: sns. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 16: f1

Type this:

echo 'aws sns create-topic --name oncall'

You will see:

aws sns create-topic --name oncall

What it means: Topic.

Example 17: f2

Type this:

echo 'Subscribe email or a chat webhook endpoint'

You will see:

Subscribe email or a chat webhook endpoint

What it means: Sub.

Example 18: f3

Type this:

echo 'Confirm the subscription email before testing'

You will see:

Confirm the subscription email before testing

What it means: Confirm.

4. Automate and verify

Step 7: Test alarm

Why: set_alarm_state. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: test. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 19: g1

Type this:

echo 'aws cloudwatch set-alarm-state --alarm-name shop-alb-5xx --state-value ALARM --state-reason test'

You will see:

aws cloudwatch set-alarm-state --alarm-name shop-alb-5xx --state-value ALARM --state-reason test

What it means: Test.

Example 20: g2

Type this:

echo 'Verify SNS delivery then set state back to OK'

You will see:

Verify SNS delivery then set state back to OK

What it means: Verify.

Example 21: g3

Type this:

echo 'Document the test in the change ticket'

You will see:

Document the test in the change ticket

What it means: Ticket.

Step 8: Composite alarms

Why: Noise control. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: composite. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 22: h1

Type this:

echo 'Composite alarms require several children before paging'

You will see:

Composite alarms require several children before paging

What it means: Composite.

Example 23: h2

Type this:

echo 'Use for multi-AZ symptoms that must agree'

You will see:

Use for multi-AZ symptoms that must agree

What it means: When.

Example 24: h3

Type this:

echo 'Keep simple alarms until you need composites'

You will see:

Keep simple alarms until you need composites

What it means: Start.

Step 9: Cost tip

Why: Custom metrics. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: cost. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 25: i1

Type this:

printf '%s\n' 'Prefer AWS standard metrics when enough' 'Custom metrics and high resolution cost more' > /tmp/cw95/COST.txt && cat /tmp/cw95/COST.txt

You will see:

Prefer AWS standard metrics when enough

What it means: Cost.

Example 26: i2

Type this:

echo 'Dashboards are cheap; alarms and API get calls add up'

You will see:

Dashboards are cheap; alarms and API get calls add up

What it means: API.

Example 27: i3

Type this:

echo 'Review unused alarms quarterly'

You will see:

Review unused alarms quarterly

What it means: Review.

5. Harden and accept

Step 10: Story MakeMyTrip

Why: Gurgaon lesson. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 28: j1

Type this:

printf '%s\n' 'At MakeMyTrip in Gurgaon, ALB 5xx had no alarm and support saw tweets first.' 'Vikram put CloudWatch alarms to SNS.' > /tmp/cw95/STORY.txt && wc -l /tmp/cw95/STORY.txt

You will see:

2 /tmp/cw95/STORY.txt

What it means: Story.

Example 29: j2

Type this:

echo 'Rough cost: about Rs 15 lakh in goodwill refunds.'

You will see:

Rough cost: about Rs 15 lakh in goodwill refunds.

What it means: Cost.

Example 30: j3

Type this:

echo 'Vikram: PASS'

You will see:

Vikram: PASS

What it means: Sign-off.

Step 11: Proof

Why: Files. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 31: k1

Type this:

ls /tmp/cw95

You will see:

NOTES.txt\nSTORY.txt\nACCEPT.txt

What it means: Tree.

Example 32: k2

Type this:

test -f /tmp/cw95/ACCEPT.txt && echo ACCEPT_OK

You will see:

ACCEPT_OK

What it means: Accept present.

Example 33: k3

Type this:

echo 'Next: OpenTelemetry tracing project 96'

You will see:

Next: OpenTelemetry tracing project 96

What it means: Bridge.

Step 12: Acceptance

Why: ALB alarm notifies SNS. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 34: z1

Type this:

echo 'Dashboard exists'

You will see:

Dashboard exists

What it means: A.

Example 35: z2

Type this:

echo 'Alarm ALARM state test'

You will see:

Alarm ALARM state test

What it means: B.

Example 36: z3

Type this:

echo 'SNS delivered'

You will see:

SNS delivered

What it means: C.

Check yourself

Q1. Who fixed MakeMyTrip tweets?

Vikram

Q2. Service?

CloudWatch

Q3. Notify?

SNS

Q4. Metric tip?

ALB 5xx

Q5. Raw count only?

no

Q6. Test?

yes

Friends, you alarmed on ALB 5xx. Did you understand? Let us try it once more. Drop the threshold in staging and prove SNS notifies, then restore.

Lab: try at home

  1. Dashboard.
  2. Alarm.
  3. SNS notify.
  4. Threshold restored.

Recap

  • Use AWS metrics first.
  • Dashboard for humans.
  • Alarm for machines.
  • SNS to on-call.
  • Test thresholds.