Ravindra BagaleCourses & study guides मराठी Track your progress

AWS SDK CloudWatch Examples: Custom Metrics, Alarms and Logs Insights (Python, PHP, Node.js)

CloudWatch is the health dashboard of AWS. From Python, PHP and Node.js we push our own business metrics, read them back, set an alarm that emails the team, write logs and search them with Logs Insights. The example: a food-delivery app with Mateo in Barcelona and Ishaan in Bengaluru watching late orders.

What you'll learn in this chapter

  • Publish custom metrics
  • Read statistics
  • Create an alarm
  • Write and filter logs
  • Run Logs Insights queries

1. The big picture: the car dashboard

Why do we need this?

Your car shows speed and fuel all the time, and a red light flashes when oil is low. Metrics are the gauges, alarms are the warning lights, logs are the trip diary you read after something went wrong.

What is it?

A metric is a Namespace + MetricName + Dimensions. PutMetricData writes points; GetMetricData reads them. PutMetricAlarm watches a statistic. Logs live in log groups and streams.

How to do it (step by step)

From event to alertApp does workPutMetricData / PutLogEventsCloudWatch aggregates per periodAlarm compares to thresholdSNS notifies the team
Batch metric data: up to 1,000 values per PutMetricData call.

2. Publish custom metrics

Why do we need this?

Mateo wants to see how many orders arrive late, by city, every minute.

What is it?

Use your own namespace, a Unit, and Dimensions to split. Keep dimension values low-cardinality (city, not order id).

How to do it (step by step)

  1. Python.
import boto3
cw = boto3.client("cloudwatch", region_name="eu-south-2")
cw.put_metric_data(Namespace="FoodApp", MetricData=[
    {"MetricName": "LateOrders", "Dimensions": [{"Name": "City", "Value": "Barcelona"}], "Value": 4, "Unit": "Count"},
    {"MetricName": "DeliveryMinutes", "Dimensions": [{"Name": "City", "Value": "Barcelona"}], "Values": [28, 31, 45], "Unit": "None"}])
print("sent")

Expected output:

sent
  1. Node.js.
import { CloudWatchClient, PutMetricDataCommand } from "@aws-sdk/client-cloudwatch";
const cw = new CloudWatchClient({ region: "ap-south-1" });
await cw.send(new PutMetricDataCommand({ Namespace: "FoodApp", MetricData: [
  { MetricName: "LateOrders", Dimensions: [{ Name: "City", Value: "Bengaluru" }], Value: 9, Unit: "Count" }] }));
console.log("sent");

Expected output:

sent
  1. PHP.
<?php
require "vendor/autoload.php";
$cw = new Aws\CloudWatch\CloudWatchClient(["region" => "eu-west-2"]);
$cw->putMetricData(["Namespace" => "FoodApp", "MetricData" => [[
  "MetricName" => "LateOrders", "Dimensions" => [["Name" => "City", "Value" => "London"]],
  "Value" => 2, "Unit" => "Count"]]]);
echo "sent", PHP_EOL;

Expected output:

sent
  1. Verify: list the metric.
aws cloudwatch list-metrics --namespace FoodApp --region ap-south-1 --query "Metrics[].Dimensions[0].Value" --output text

Expected output:

Bengaluru

3. Read statistics

Why do we need this?

Ishaan builds a small report: late orders per 5 minutes for the last hour.

What is it?

GetMetricData takes MetricDataQueries with a Stat and Period and returns Timestamps and Values arrays.

How to do it (step by step)

  1. Python.
from datetime import datetime, timedelta, timezone
now = datetime.now(timezone.utc)
r = cw.get_metric_data(StartTime=now - timedelta(hours=1), EndTime=now, MetricDataQueries=[{
    "Id": "late", "MetricStat": {"Stat": "Sum", "Period": 300, "Metric": {
        "Namespace": "FoodApp", "MetricName": "LateOrders",
        "Dimensions": [{"Name": "City", "Value": "Barcelona"}]}}}])
print(sum(r["MetricDataResults"][0]["Values"]))

Expected output:

4.0
  1. Node.js.
import { GetMetricDataCommand } from "@aws-sdk/client-cloudwatch";
const end = new Date(), start = new Date(end - 3600e3);
const r = await cw.send(new GetMetricDataCommand({ StartTime: start, EndTime: end, MetricDataQueries: [{
  Id: "late", MetricStat: { Stat: "Sum", Period: 300, Metric: { Namespace: "FoodApp", MetricName: "LateOrders",
  Dimensions: [{ Name: "City", Value: "Bengaluru" }] } } }] }));
console.log(r.MetricDataResults[0].Values);

Expected output:

[ 9 ]
  1. PHP: math across metrics.
<?php
$r = $cw->getMetricData(["StartTime" => strtotime("-1 hour"), "EndTime" => time(), "MetricDataQueries" => [
  ["Id" => "l", "ReturnData" => false, "MetricStat" => ["Stat" => "Sum", "Period" => 3600,
    "Metric" => ["Namespace" => "FoodApp", "MetricName" => "LateOrders", "Dimensions" => [["Name" => "City", "Value" => "London"]]]]],
  ["Id" => "pct", "Expression" => "l * 100 / 50", "Label" => "late % of 50 orders"]]]);
echo $r["MetricDataResults"][0]["Values"][0], "%", PHP_EOL;

Expected output:

4%
  1. Verify with the CLI.
aws cloudwatch get-metric-statistics --namespace FoodApp --metric-name LateOrders --dimensions Name=City,Value=Bengaluru --statistics Sum --period 3600 --start-time $(date -u -d "-1 hour" +%FT%TZ) --end-time $(date -u +%FT%TZ) --query "Datapoints[0].Sum"

Expected output:

9.0

4. Create an alarm

Why do we need this?

If more than 20 orders are late in 5 minutes in Bengaluru, the on-call team must get an email.

What is it?

PutMetricAlarm sets threshold, period, evaluation periods and AlarmActions (an SNS topic). States: OK, ALARM, INSUFFICIENT_DATA.

How to do it (step by step)

  1. Python.
cw.put_metric_alarm(AlarmName="late-orders-blr", Namespace="FoodApp", MetricName="LateOrders",
    Dimensions=[{"Name": "City", "Value": "Bengaluru"}], Statistic="Sum", Period=300,
    EvaluationPeriods=1, Threshold=20, ComparisonOperator="GreaterThanThreshold",
    TreatMissingData="notBreaching",
    AlarmActions=["arn:aws:sns:ap-south-1:123456789012:oncall"])
print("alarm ok")

Expected output:

alarm ok
  1. Node.js: force the state to test the email.
import { SetAlarmStateCommand } from "@aws-sdk/client-cloudwatch";
await cw.send(new SetAlarmStateCommand({ AlarmName: "late-orders-blr", StateValue: "ALARM", StateReason: "test" }));
console.log("forced");

Expected output:

forced  (on-call gets an email)
  1. PHP: read alarm state.
<?php
$cw = new Aws\CloudWatch\CloudWatchClient(["region" => "ap-south-1"]);
$a = $cw->describeAlarms(["AlarmNames" => ["late-orders-blr"]])["MetricAlarms"][0];
echo $a["StateValue"], " ", $a["Threshold"], PHP_EOL;

Expected output:

ALARM 20
  1. Verify: alarm history.
aws cloudwatch describe-alarm-history --alarm-name late-orders-blr --history-item-type StateUpdate --max-items 1 --query "AlarmHistoryItems[0].HistorySummary" --output text

Expected output:

Alarm updated from OK to ALARM

5. Logs and Logs Insights

Why do we need this?

A rider in Paris says order PA-311 never arrived; Mateo searches the logs.

What is it?

Create a group and stream, PutLogEvents with millisecond timestamps, FilterLogEvents for simple search, StartQuery/GetQueryResults for Logs Insights.

How to do it (step by step)

  1. Python: write JSON log lines.
import json, time
logs = boto3.client("logs", region_name="eu-west-3")
logs.create_log_group(logGroupName="/foodapp/orders")
logs.create_log_stream(logGroupName="/foodapp/orders", logStreamName="api-1")
logs.put_log_events(logGroupName="/foodapp/orders", logStreamName="api-1", logEvents=[
    {"timestamp": int(time.time() * 1000), "message": json.dumps({"order": "PA-311", "status": "picked", "total": "€23.40"})}])
print("logged")

Expected output:

logged
  1. Node.js: filter events.
import { CloudWatchLogsClient, FilterLogEventsCommand } from "@aws-sdk/client-cloudwatch-logs";
const logs = new CloudWatchLogsClient({ region: "eu-west-3" });
const r = await logs.send(new FilterLogEventsCommand({ logGroupName: "/foodapp/orders", filterPattern: '{ $.order = "PA-311" }' }));
for (const e of r.events) console.log(e.message);

Expected output:

{"order": "PA-311", "status": "picked", "total": "€23.40"}
  1. PHP: Logs Insights query.
<?php
$logs = new Aws\CloudWatchLogs\CloudWatchLogsClient(["region" => "eu-west-3"]);
$q = $logs->startQuery(["logGroupName" => "/foodapp/orders", "startTime" => time() - 3600, "endTime" => time(),
  "queryString" => "stats count(*) by status"])["queryId"];
do { sleep(1); $r = $logs->getQueryResults(["queryId" => $q]); } while ($r["status"] === "Running");
foreach ($r["results"] as $row) echo $row[0]["value"], "=", $row[1]["value"], PHP_EOL;

Expected output:

picked=1
  1. Verify: tail the group.
aws logs tail /foodapp/orders --since 10m --region eu-west-3 | grep -c PA-311

Expected output:

1

6. The same call in Java, Go, .NET and the CLI

Why do we need this?

Teams often mix languages. Once you know the call in Python, PHP and Node.js, the other SDKs are the same API with different syntax.

What is it?

All of them call the PutMetricData API. Use the CLI line to test it quickly from a terminal.

How to do it (step by step)

StackCode
AWS CLIaws cloudwatch put-metric-data --namespace FoodApp --metric-name LateOrders --dimensions City=Bengaluru --value 9 --unit Count
Java v2cw.putMetricData(b -> b.namespace("FoodApp").metricData(MetricDatum.builder().metricName("LateOrders").value(9.0).build()));
Go v2client.PutMetricData(ctx, &cloudwatch.PutMetricDataInput{Namespace: aws.String("FoodApp"), MetricData: []types.MetricDatum{{MetricName: aws.String("LateOrders"), Value: aws.Float64(9)}}})
.NETawait cw.PutMetricDataAsync(new PutMetricDataRequest { Namespace = "FoodApp", MetricData = { new MetricDatum { MetricName = "LateOrders", Value = 9 } } });

7. Clean up so nothing keeps costing

Why do we need this?

Lab resources left running cost money and can be a security risk.

What is it?

Delete in reverse order: children first, then parents.

How to do it (step by step)

  1. Delete the alarm and log group (metrics expire by themselves).
aws cloudwatch delete-alarms --alarm-names late-orders-blr --region ap-south-1
aws logs delete-log-group --log-group-name /foodapp/orders --region eu-west-3

Expected output:

(no output means success)

Ravindra Bagale's Tip

Set a retention period on every log group (PutRetentionPolicy); the default keeps logs, and the bill, forever.

8. Interview questions and answers

Q. What identifies a metric?

Namespace, MetricName and its Dimensions.

Q. GetMetricData vs GetMetricStatistics?

GetMetricData is newer, batches many queries and supports metric math.

Q. Alarm states?

OK, ALARM and INSUFFICIENT_DATA.

Q. How do you test an alarm action?

SetAlarmState to ALARM temporarily.

Q. Simple search vs Logs Insights?

FilterLogEvents matches patterns; Logs Insights runs a query language with stats and sort.

Try at home

  1. Publish DeliveryMinutes from Node.js and alarm when p90 > 40.
  2. Write a Python Logs Insights query that lists the 5 slowest orders.

In short

  1. PutMetricData with dimensions.
  2. GetMetricData + math.
  3. PutMetricAlarm → SNS.
  4. PutLogEvents in ms.
  5. Logs Insights StartQuery.

Practice Tasks

About this scenario

Goa Beach Stays is a hospitality business in Panaji. Rosario Fernandes looks after its IT and cloud work. They count money in INR and run their AWS resources in the ap-south-1 Region. In the tasks below you do real jobs for them using what this lesson teaches. All names, account IDs, IP addresses and numbers are made up for practice.

0 / 12

Easy List metrics in a namespace

Why: See which metrics AWS already collects for a service.

What to do: List the metrics in the AWS/EC2 namespace.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Easy Create an SNS topic

Why: A topic is a channel: publish once, every subscriber gets it.

What to do: Create topic goabeachstays-alerts.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Easy List alarms that are firing

Why: In an incident, first see which alarms are in ALARM state.

What to do: Describe only the alarms whose state is ALARM.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Easy Reboot an instance

Why: A reboot keeps the same disks and IPs, unlike stop/start which may change the public IP.

What to do: Reboot the instance i-03a3c4d31b3111a2b.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Easy Publish to one topic

Why: An alerting app only needs to publish.

What to do: Allow sns:Publish on arn:aws:sns:ap-south-1:113713948363:goabeachstays-alerts only.

How:
  1. Start with { "Version": "2012-10-17", "Statement": [ ... ] }.
  2. In each statement write Effect, Action and Resource.
  3. Give only what the task needs (least privilege).
  4. Use double quotes and commas correctly. Click Check answer.

Medium Publish a custom metric

Why: Custom metrics let you graph and alarm on business numbers, like orders per minute.

What to do: Publish metric OrdersPlaced with value 12 in namespace GoaBeachStays/Shop.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Medium Subscribe an email

Why: Email subscribers must click the confirm link before they receive messages.

What to do: Subscribe ops@goabeachstays.in by email to arn:aws:sns:ap-south-1:113713948363:goabeachstays-alerts.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Medium Check target health

Why: If targets are unhealthy the site shows 502/503 errors.

What to do: Show the health of the targets in arn:aws:elasticloadbalancing:ap-south-1:113713948363:targetgroup/goabeachstays-tg/6d0ecf831eec9f09.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Medium Let a function write logs

Why: Without these actions, Lambda logs never appear in CloudWatch.

What to do: Allow logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents on arn:aws:logs:*:*:* only.

How:
  1. Start with { "Version": "2012-10-17", "Statement": [ ... ] }.
  2. In each statement write Effect, Action and Resource.
  3. Give only what the task needs (least privilege).
  4. Use double quotes and commas correctly. Click Check answer.

Medium Kubernetes LoadBalancer Service

Why: A Service of type LoadBalancer gets an AWS load balancer.

What to do: Write a Service of type LoadBalancer selecting app: web, port 80 to targetPort 8080.

How:
  1. Start from the top-level keys the task names.
  2. Indent child keys with 2 spaces (no tabs).
  3. Fill in each property with the value from the task.
  4. Click Check answer. Key order does not matter.

Hard Create a CPU alarm

Why: An alarm watches a metric and notifies you before users notice a problem.

What to do: Create alarm goabeachstays-cpu-high: AWS/EC2 CPUUtilization, Average over 300 seconds, 1 evaluation period, GreaterThanThreshold 80, for InstanceId=i-03a3c4d31b3111a2b, notify arn:aws:sns:ap-south-1:113713948363:goabeachstays-alerts.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Hard Auto-recover an instance with an alarm

Why: A recover action moves the instance to healthy hardware when the system check fails.

What to do: Create alarm goabeachstays-recover on AWS/EC2 StatusCheckFailed_System, Maximum over 60 s, 2 periods, GreaterThanOrEqualToThreshold 1, InstanceId=i-03a3c4d31b3111a2b, action arn:aws:automate:ap-south-1:ec2:recover.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Revision, comparison and interview cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

CloudWatch metrics. PutMetricData: namespace + name. Dimensions split by city. Up to 1000 values per call. Read with GetMetricData.
Revision card: CloudWatch metrics · Download PNG · Open card page
CloudWatch alarms. PutMetricAlarm on a threshold. States: OK, ALARM, INSUFFICIENT. Action: an SNS topic ARN. Logs Insights: StartQuery.
Revision card: CloudWatch alarms · Download PNG · Open card page
Metrics vs Logs. Numbers over time. / Text lines of events.
Comparison: Metrics vs Logs · Download PNG · Open card page
Alarm stuck in INSUFFICIENT_DATA?. Dimensions do not match exactly. Period longer than data rate. Set TreatMissingData.
Interview card: Alarm stuck in INSUFFICIENT_DATA? · Download PNG · Open card page

More cards: Ravi Sir's Cards.