How to Send Logs from Multiple EC2 Instances to CloudWatch Logs (Agent, SSM, Logs Insights)

By Ravindra Bagale, trainer · LinkedIn · Last updated: 21 October 2026

1. A story: five servers, one error

A food-delivery app runs on 5 web servers behind a load balancer. A customer says "payment failed at 8:14 PM". Which server served him? Without central logs you SSH into 5 servers and run grep 5 times. If Auto Scaling already terminated that server, its logs are gone for ever.

With CloudWatch Logs, all 5 servers send their logs to one log group. One query finds the error in seconds, even from a server that no longer exists.

2. Log groups, streams and retention

Why do we need this?

Millions of lines need folders, or you can never find anything. Old logs also cost money, so we must delete them on time.

What is it?

• Log event: one line with a time. Example: one nginx request.
• Log stream: all lines from one source. We use one stream per instance, named by instance ID.
• Log group: a folder of streams of the same kind. Example: /app/nginx/access holds the access logs of every web server.
• Retention: how many days a log group keeps data. Default is Never expire, which grows the bill for ever. Set it, for example 30 days for access logs, 90 or more for audit logs.

3. Step 1: the IAM role

Why do we need this?

The agent on the server must prove to AWS that it is allowed to write logs. We never put access keys on servers: keys leak. A role gives short-lived keys automatically.

What is it?

CloudWatchAgentServerPolicy is an AWS managed policy. It allows the agent to send metrics and logs (PutMetricData, CreateLogStream, PutLogEvents and so on) and to read its config from Parameter Store. AmazonSSMManagedInstanceCore lets Systems Manager manage the server (we use it in Step 5).

How to do it (step by step)

1. Open IAM → Roles → Create role.
2. Trusted entity type: AWS service. Use case: EC2. Next.
3. Search and tick CloudWatchAgentServerPolicy.
4. Search and tick AmazonSSMManagedInstanceCore. Next.
5. Role name: EC2-CloudWatchAgent-Role. Create role.
6. EC2 → select each server → Actions → Security → Modify IAM role → choose EC2-CloudWatchAgent-Role → Update IAM role.
7. Verify on the server:

TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/info
# expected: "InstanceProfileArn" : "arn:aws:iam::...:instance-profile/EC2-CloudWatchAgent-Role"

4. Step 2: install the agent (one server)

1. SSH into web1 (Amazon Linux 2023) and install nginx so we have logs:

sudo dnf install -y nginx amazon-cloudwatch-agent
sudo systemctl enable --now nginx
curl -s -o /dev/null -w "%{http_code}\n" http://localhost/   # expected: 200

On Ubuntu: download https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb and run sudo dpkg -i -E ./amazon-cloudwatch-agent.deb. On Windows use the MSI from the same bucket.

5. Step 3: configure with the wizard

Why do we need this?

The config file has many options. The wizard asks simple questions and writes a correct file for you. Good for the first server.

How to do it (step by step)

1. Run sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard.
2. OS: linux. EC2 or on-premises: EC2. User: cwagent. Why cwagent: the agent should not run as root.
3. StatsD: no. CollectD: no. Why: we do not use them; collectd also needs a package, or the agent fails to start.
4. Monitor host metrics: yes. CPU per core: no. EC2 dimensions: yes. Aggregate: yes. Resolution: 60s. Level: Basic (mem and disk).
5. Existing config: no. Monitor log files: yes.
6. Log file path: /var/log/nginx/access.log. Log group: /app/nginx/access. Log group class: STANDARD. Stream name: {instance_id}. Retention: 30.
7. Add another: /var/log/nginx/error.log → /app/nginx/error, same stream name and retention. Then no more files.
8. Store config in SSM Parameter Store: yes, name AmazonCloudWatch-linux. Why: other servers can download the same config in Step 5.
9. The file is saved at /opt/aws/amazon-cloudwatch-agent/bin/config.json.

6. Step 4: the JSON file explained, and start the agent

This is the same config written by hand. Each part has a reason:

{
  "agent": { "metrics_collection_interval": 60, "run_as_user": "cwagent" },
  "metrics": {
    "namespace": "CWAgent",
    "append_dimensions": { "InstanceId": "${aws:InstanceId}" },
    "metrics_collected": {
      "mem":  { "measurement": ["mem_used_percent"] },
      "disk": { "measurement": ["used_percent"], "resources": ["/"] }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          { "file_path": "/var/log/nginx/access.log",
            "log_group_name": "/app/nginx/access",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 30 },
          { "file_path": "/var/log/nginx/error.log",
            "log_group_name": "/app/nginx/error",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 30 }
        ]
      }
    }
  }
}

1. Start the agent with this file:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 -s \
  -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
# expected: "status": "running"
# from Parameter Store instead of a file:  -c ssm:AmazonCloudWatch-linux
tail -n 20 /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log   # look for errors

2. Make some traffic: for i in $(seq 1 50); do curl -s -o /dev/null http://localhost/; curl -s -o /dev/null http://localhost/nope; done.
3. Verify: CloudWatch → Log groups → /app/nginx/access → a stream named like i-0abc… with your requests (200 and 404).
4. Verify metrics: CloudWatch → Metrics → CWAgent → InstanceId → mem_used_percent.

Log entries for other stacks (add to collect_list):

{ "file_path": "/home/ec2-user/.pm2/logs/*-out.log", "log_group_name": "/app/node",   "log_stream_name": "{instance_id}", "retention_in_days": 14 },
{ "file_path": "/opt/app/logs/spring.log",           "log_group_name": "/app/java",   "log_stream_name": "{instance_id}", "retention_in_days": 30,
  "multi_line_start_pattern": "^\\d{4}-\\d{2}-\\d{2}" },
{ "file_path": "/var/log/gunicorn/error.log",        "log_group_name": "/app/python", "log_stream_name": "{instance_id}", "retention_in_days": 30 },
{ "file_path": "/var/log/httpd/access_log",          "log_group_name": "/app/httpd/access", "log_stream_name": "{instance_id}", "retention_in_days": 30 }

Why multi_line_start_pattern for Java: a Java stack trace is 30 lines. Without it, every line becomes a separate event and the error is hard to read. With it, a new event starts only at a line that begins with a date.

7. Step 5: many servers with Systems Manager

Why do we need this?

Doing Steps 2-4 by SSH on 20 servers is slow and each server may end up a little different. Systems Manager (SSM) runs the same command on all of them, with no SSH and no open port 22.

How to do it (step by step)

1. Give every server the role EC2-CloudWatchAgent-Role (Step 1) and a tag Role = web.
2. Open Systems Manager → Fleet Manager. Verify all servers appear as Online. (The SSM agent is pre-installed on Amazon Linux and Ubuntu AMIs; it needs outbound HTTPS.)
3. Open Systems Manager → Run Command → Run command. Document: AWS-ConfigureAWSPackage.
4. Action: Install. Name: AmazonCloudWatchAgent. Version: latest.
5. Targets: Specify instance tags → Role = web. Why tags: new web servers are included without editing the command.
6. Output options: untick S3 for the lab. Run. Verify: each target shows Success.
7. Run command again. Document: AmazonCloudWatch-ManageAgent. Action configure, Mode ec2, Optional Configuration Source ssm, Optional Configuration Location AmazonCloudWatch-linux, Optional Restart yes. Same tag target. Run.
8. Verify: the log group /app/nginx/access now has one stream per server.

The same with the CLI:

aws ssm put-parameter --name AmazonCloudWatch-linux --type String --overwrite \
  --value file://config.json
aws ssm send-command --document-name AWS-ConfigureAWSPackage \
  --targets Key=tag:Role,Values=web \
  --parameters '{"action":["Install"],"name":["AmazonCloudWatchAgent"]}'
aws ssm send-command --document-name AmazonCloudWatch-ManageAgent \
  --targets Key=tag:Role,Values=web \
  --parameters '{"action":["configure"],"mode":["ec2"],"optionalConfigurationSource":["ssm"],"optionalConfigurationLocation":["AmazonCloudWatch-linux"],"optionalRestart":["yes"]}'
aws logs describe-log-streams --log-group-name /app/nginx/access --query 'logStreams[].logStreamName'
# expected: ["i-0aaa...", "i-0bbb...", "i-0ccc..."]

8. Step 6: retention

1. CloudWatch → Log groups → tick /app/nginx/access → Actions → Edit retention setting(s) → 1 month (30 days) → Save.
2. CLI: aws logs put-retention-policy --log-group-name /app/nginx/error --retention-in-days 30.
3. Verify: the Retention column shows 1 month instead of Never expire.

9. Step 7: search everything with Logs Insights

Why do we need this?

Scrolling through streams is slow. Logs Insights is a small query language that searches all streams of one or more log groups at once.

How to do it (step by step)

1. CloudWatch → Logs Insights. Select log group /app/nginx/access. Time range: 1h.
2. Paste a query below and click Run query. Start with the simplest.

# 1. latest 20 lines from all servers
fields @timestamp, @logStream, @message
| sort @timestamp desc
| limit 20

# 2. only 404 and 5xx lines
fields @timestamp, @logStream, @message
| filter @message like / 404 / or @message like / 50[0-9] /
| sort @timestamp desc

# 3. requests per server (which server gets the most?)
stats count(*) as requests by @logStream
| sort requests desc

# 4. errors per 5 minutes, ready for a graph
filter @message like / 50[0-9] /
| stats count(*) as errors by bin(5m)

# 5. top client IPs (nginx combined format: IP is the first word)
parse @message "* - * [*] \"* * *\" * *" as ip, user, time, method, path, proto, status, bytes
| stats count(*) as hits by ip
| sort hits desc
| limit 10

Verify: query 3 shows one row per instance ID, query 2 shows the /nope 404 lines you created.

Bonus: alarm on errors in logs (metric filter)

1. Log group /app/nginx/access → Actions → Create metric filter.
2. Filter pattern: [ip, id, user, ts, request, status=5*, size]. Test it, Next.
3. Filter name nginx-5xx, namespace App/Web, metric name Nginx5xx, value 1, default value 0. Create.
4. Create an alarm on App/Web Nginx5xx, statistic Sum, 5 minutes, > 10, SNS ops-alerts.

10. Cleanup

1. Delete the log groups /app/nginx/access and /app/nginx/error (Actions → Delete log group(s)).
2. Delete the metric filter alarm and the parameter: aws ssm delete-parameter --name AmazonCloudWatch-linux.
3. Stop the agent (amazon-cloudwatch-agent-ctl -a stop) or terminate the servers. Why: the agent keeps sending custom metrics, which are billed.

11. Interview questions

Revision, comparison and interview cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

More cards: Ravi Sir's Cards.