Chapter 58: Docker Projects Part 2: AWS RDS, DynamoDB, S3, Reel Upload, Microservices, Queues, Logging, Monitoring and Cloud Deploys

58.11 Project 27: Prometheus, Grafana and cAdvisor monitoring for the microservices shop

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

In Project 26 we collected logs. Logs tell us what happened to one request. But some questions need numbers over time. Is the payments container using more and more memory every hour? How many requests per second hit the gateway? What percent of them fail? Is the server disk almost full? At Flipkart-style companies, these numbers are called metrics. A dashboard shows them, and an alert wakes up the on-call engineer before customers notice. The most common free stack is Prometheus and Grafana. Prometheus pulls (scrapes) numbers every 15 seconds and stores them. Grafana draws them. Alertmanager sends alerts. For Docker we add cAdvisor, which gives CPU, memory and network for every container, and Node Exporter for the host. The gateway nginx gets an exporter too, and our Node.js service exposes its own request metrics. We run everything with Compose, build a dashboard, and make an alert fire on purpose.

What: every term in plain English

• Metric — A number with a name and labels, measured over time, like container_memory_usage_bytes{name="shop-payments-1"}.
• Scrape — Prometheus calls an HTTP /metrics page every interval and saves the numbers.
• Exporter — A small program that turns something (a host, nginx, MySQL) into a /metrics page.
• cAdvisor — Google's container advisor. Exports CPU, memory, network and disk for every container.
• Node Exporter — Exports host CPU, memory, disk and network.
• PromQL — Prometheus query language, for example rate(http_requests_total[5m]).
• Alert rule — A PromQL condition with a for: time. When true long enough, the alert fires.
• Alertmanager — Groups alerts and sends them to email, Slack or a webhook.

How: step by step

1. Add metrics to one service. prom-client in the Node users service exposes /metrics.
2. Start the monitoring stack. Prometheus, Grafana, cAdvisor, Node Exporter, nginx exporter, Alertmanager.
3. Check targets. All targets UP in the Prometheus UI.
4. Build dashboards. Provision a data source and import ready dashboards in Grafana.
5. Fire an alert. Stress a container and see the alert go from pending to firing.
6. Clean up. down -v.

Architecture

Folder layout

project/
  shop/                       (from Project 21)
    users/metrics.js  gateway/status.conf
  monitoring/
    compose.yaml
    prometheus/prometheus.yml  prometheus/alerts.yml
    alertmanager/alertmanager.yml
    grafana/provisioning/datasources/prom.yaml
    queries.txt  cleanup.sh

Example 1: Application metrics in the Node

Goal. Application metrics in the Node.js users service, and nginx status for the gateway.

shop/users/metrics.js

const client = require("prom-client");
client.collectDefaultMetrics({ prefix: "users_" });

const httpRequests = new client.Counter({
  name: "http_requests_total",
  help: "HTTP requests by route and status",
  labelNames: ["route", "method", "status"]
});
const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "Request time in seconds",
  labelNames: ["route", "method"],
  buckets: [0.01, 0.05, 0.1, 0.3, 1, 3]
});

function middleware(req, res, next) {
  const end = httpDuration.startTimer({ method: req.method });
  res.on("finish", () => {
    const route = req.route ? req.route.path : "unknown";
    httpRequests.inc({ route, method: req.method, status: res.statusCode });
    end({ route });
  });
  next();
}

async function handler(req, res) {
  res.set("Content-Type", client.register.contentType);
  res.end(await client.register.metrics());
}

module.exports = { middleware, handler };

shop/users/server-snippet.js

const metrics = require("./metrics");
app.use(metrics.middleware);
app.get("/metrics", metrics.handler);

shop/gateway/status.conf

server {
    listen 8081;
    location = /stub_status {
        stub_status;
        allow 172.16.0.0/12;
        deny all;
    }
}

What each line does

• Counter — Only goes up. We use rate() on it to get requests per second.
• Histogram + buckets — Counts how many requests were faster than 10 ms, 50 ms and so on. Gives p95 latency.
• route label, not URL — /users/:id is one label value. Using the real URL with ids makes millions of series and crashes Prometheus.
• collectDefaultMetrics — Node.js memory, event loop lag and GC numbers for free.
• stub_status on 8081 — nginx shows active connections and request totals. Only Docker networks may read it.
• Add prom-client — Add "prom-client": "15.1.3" to users/package.json and rebuild.

Green output

$ docker compose up -d --build users gateway
$ docker compose exec users wget -qO- localhost:3000/metrics | grep http_requests_total | head -2
# HELP http_requests_total HTTP requests by route and status
http_requests_total{route="/users/:id",method="GET",status="200"} 12

Expected result. The users service now has its own metrics page. Next, Prometheus will scrape it.

Example 2: The monitoring stack: Prometheus, exporters, cAdvisor, Alertmanager and Grafana

Goal. The monitoring stack: Prometheus, exporters, cAdvisor, Alertmanager and Grafana.

monitoring/compose.yaml

services:
  prometheus:
    image: prom/prometheus:v2.54.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=15d
      - --web.enable-lifecycle
    volumes:
      - ./prometheus:/etc/prometheus:ro
      - prom-data:/prometheus
    ports: ["127.0.0.1:9090:9090"]
    networks: [shop, mon]

  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes: [./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro]
    ports: ["127.0.0.1:9093:9093"]
    networks: [mon]

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    privileged: true
    devices: ["/dev/kmsg"]
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks: [mon]

  node-exporter:
    image: prom/node-exporter:v1.8.2
    command: ["--path.rootfs=/host"]
    pid: host
    volumes: ["/:/host:ro,rslave"]
    networks: [mon]

  nginx-exporter:
    image: nginx/nginx-prometheus-exporter:1.3.0
    command: ["--nginx.scrape-uri=http://gateway:8081/stub_status"]
    networks: [shop, mon]

  grafana:
    image: grafana/grafana:11.2.2
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:-change-me-grafana}
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    ports: ["127.0.0.1:3001:3000"]
    networks: [mon]

networks:
  mon: {}
  shop:
    external: true
    name: shop_default

volumes:
  prom-data:
  grafana-data:

monitoring/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files: [alerts.yml]

alerting:
  alertmanagers:
    - static_configs: [{ targets: ["alertmanager:9093"] }]

scrape_configs:
  - job_name: prometheus
    static_configs: [{ targets: ["localhost:9090"] }]
  - job_name: cadvisor
    static_configs: [{ targets: ["cadvisor:8080"] }]
  - job_name: node
    static_configs: [{ targets: ["node-exporter:9100"] }]
  - job_name: nginx
    static_configs: [{ targets: ["nginx-exporter:9113"] }]
  - job_name: users
    dns_sd_configs:
      - names: ["users"]
        type: A
        port: 3000

monitoring/grafana/provisioning/datasources/prom.yaml

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

What each line does

• shop_default external — Prometheus and the nginx exporter join the shop network, so they can reach users and gateway by name.
• retention 15d — Metrics are kept 15 days on the prom-data volume.
• cAdvisor privileged and mounts — It reads cgroups and Docker data from the host to measure each container. Read-only mounts.
• pid: host for node-exporter — It sees host processes and the real root filesystem.
• dns_sd_configs — Prometheus asks Docker DNS for users and scrapes every copy, even after --scale.
• Grafana on 127.0.0.1:3001 — Only local access. Grafana port 3000 inside, 3001 outside to avoid clashes.
• Provisioned data source — Grafana starts with Prometheus already connected. No clicking.

Green output

$ cd monitoring
$ docker compose up -d
 Container monitoring-prometheus-1  Started
 Container monitoring-cadvisor-1  Started
 Container monitoring-grafana-1  Started
$ curl -s localhost:9090/api/v1/targets | grep -o '"job":"[a-z]*","health":"[a-z]*"' | sort -u
"job":"cadvisor","health":"up"
"job":"nginx","health":"up"
"job":"node","health":"up"
"job":"prometheus","health":"up"
"job":"users","health":"up"

Expected result. Every target is up. Prometheus now collects container, host, nginx and app numbers.

Example 3: Alert rules, Alertmanager, PromQL queries and making an alert fire

Goal. Alert rules, Alertmanager, PromQL queries and making an alert fire.

monitoring/prometheus/alerts.yml

groups:
  - name: shop
    rules:
      - alert: ContainerDown
        expr: time() - container_last_seen{name=~"shop-.+"} > 60
        for: 1m
        labels: { severity: critical }
        annotations:
          summary: "Container {{ $labels.name }} is down"

      - alert: ContainerHighCPU
        expr: sum by (name) (rate(container_cpu_usage_seconds_total{name=~"shop-.+"}[1m])) > 0.8
        for: 2m
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.name }} CPU above 80 percent for 2 minutes"

      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "More than 5 percent of user requests fail"

      - alert: DiskAlmostFull
        expr: (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 > 90
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Root disk has less than 10 percent free"

monitoring/alertmanager/alertmanager.yml

route:
  receiver: team
  group_by: [alertname]
  group_wait: 30s
  repeat_interval: 4h
receivers:
  - name: team
    webhook_configs:
      - url: http://alert-sink:8080/hook
        send_resolved: true

monitoring/queries.txt

Requests per second, by route:
  sum by (route) (rate(http_requests_total[1m]))
p95 latency in seconds:
  histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
Memory per shop container in MB:
  container_memory_working_set_bytes{name=~"shop-.+"} / 1024 / 1024
nginx active connections:
  nginx_connections_active

monitoring/cleanup.sh

cd "$(dirname "$0")"
docker compose down -v
docker rm -f shop-stress-1 2>/dev/null
echo "monitoring removed"

What each line does

• for: 2m — The condition must stay true for 2 minutes. Short spikes do not wake anyone.
• rate(...[1m]) — CPU seconds per second. 0.8 means 80 percent of one core.
• Error ratio — 5xx requests divided by all requests. Better than a raw error count.
• DiskAlmostFull — Used percent = (1 - free / size) * 100. Above 90 used means under 10 percent free.
• Webhook receiver — Alertmanager posts JSON to a URL. Swap it for email_configs or slack_configs with your own webhook URL from a secret.
• Grafana dashboards — Import dashboard IDs 14282 (cAdvisor) and 1860 (Node Exporter Full) from grafana.com in Dashboards, Import.

Green output

$ curl -s -X POST localhost:9090/-/reload
$ docker run -d --name shop-stress-1 --network shop_default alpine sh -c 'while true; do :; done'
$ sleep 150; curl -s localhost:9090/api/v1/alerts | grep -o '"alertname":"[A-Za-z]*","name":"[a-z0-9-]*"},"state":"[a-z]*"'
"alertname":"ContainerHighCPU","name":"shop-stress-1"},"state":"firing"
$ curl -s localhost:9093/api/v2/alerts | grep -o '"alertname":"[A-Za-z]*"'
"alertname":"ContainerHighCPU"
$ docker rm -f shop-stress-1

Expected result. A busy container made the CPU alert fire, and Alertmanager received it. Removing it resolves the alert.

Every command, step by step (plain docker and Compose)

1. Add metrics to users

   docker compose up -d --build users gateway
   

   What it does: Rebuild with prom-client and stub_status.

   Container shop-users-1 Started
   

2. Check /metrics

   docker compose exec users wget -qO- localhost:3000/metrics | head
   

   What it does: Metrics text format.

   # HELP users_process_cpu_user_seconds_total
   

3. Start monitoring

   cd monitoring && docker compose up -d
   

   What it does: Prometheus, exporters, Alertmanager and Grafana.

   Container monitoring-grafana-1 Started
   

4. Check targets

   http://localhost:9090/targets
   

   What it does: All jobs UP.

   5 of 5 up
   

5. Try PromQL

   In Prometheus Graph: sum by (route) (rate(http_requests_total[1m]))
   

   What it does: Requests per second per route.

   {route="/users/:id"} 0.4
   

6. Container memory

   container_memory_working_set_bytes{name=~"shop-.+"} / 1024 / 1024
   

   What it does: MB used by each shop container.

   shop-orders-1 61.2
   

7. Open Grafana

   http://localhost:3001 (admin / change-me-grafana)
   

   What it does: Prometheus data source is already set.

   Data source: Prometheus (default)
   

8. Import dashboards

   Dashboards, New, Import, ID 14282 then 1860
   

   What it does: Container and host dashboards.

   Dashboard imported
   

9. Make load

   for i in $(seq 1 300); do curl -s -o /dev/null localhost:8080/api/users/1; done
   

   What it does: Graphs move.

   (no output)
   

10. Fire an alert

    docker run -d --name shop-stress-1 --network shop_default alpine sh -c 'while true; do :; done'
    

    What it does: CPU 100 percent.

    alert state firing after about 2 minutes
    

11. Stop the stress

    docker rm -f shop-stress-1
    

    What it does: Alert resolves.

    shop-stress-1
    

12. Clean up

    bash monitoring/cleanup.sh
    

    What it does: Removes monitoring containers and volumes.

    monitoring removed
    

When it is red 1

Get "http://users:3000/metrics": dial tcp: lookup users on 127.0.0.11:53: no such host

Cause. Prometheus is not on the shop network.

Fix, step by step

1. Add the shop network as external with the real name (docker network ls).
2. Restart Prometheus.

When it is red 2

Failed to start container manager: inotify_add_watch /sys/fs/cgroup/...: no such file or directory

Cause. cAdvisor cannot read cgroups, often on Docker Desktop or cgroup v2 hosts without mounts.

Fix, step by step

1. Keep privileged and the /sys mount.
2. Use the newest cAdvisor version.

When it is red 3

Error loading config: yaml: unmarshal errors

Cause. A YAML indentation mistake in prometheus.yml or alerts.yml.

Fix, step by step

1. Check with: docker run --rm -v $PWD/prometheus:/p --entrypoint promtool prom/prometheus:v2.54.1 check config /p/prometheus.yml

When it is red 4

Grafana: Post "http://localhost:9090/api/v1/query": connection refused

Cause. The data source URL uses localhost, which is the Grafana container itself.

Fix, step by step

1. Use http://prometheus:9090.

When it is red 5

Prometheus memory keeps growing

Cause. High cardinality: a label with many values, such as full URLs or user ids.

Fix, step by step

1. Use route patterns as labels.
2. Check with: topk(10, count by (name) ({name=~".+"}))

When it is red 6

Alert stays pending and never fires

Cause. The condition is not true for the whole for: period.

Fix, step by step

1. Lower for: in practice, or keep the stress running longer.

Practice tasks

http://localhost:9090/targets

container_memory_working_set_bytes

Dashboards, Import

Counter('orders_created_total', ...)

--scale users=3

docker run ... while true

docker stop shop-orders-1

histogram_quantile(0.95, ...)

le="0.3"

bash monitoring/cleanup.sh

Recap

• Metrics are numbers over time; Prometheus scrapes them from /metrics pages.
• cAdvisor covers containers, Node Exporter covers the host, exporters cover nginx and databases.
• Instrument your own app with counters and histograms, using low-cardinality labels.
• Alert rules with for: avoid noise; Alertmanager routes alerts to people.
• Grafana with a provisioned data source and imported dashboards gives views in minutes.
• Keep monitoring ports on 127.0.0.1 or behind a login.

FAQ

Q. Prometheus or CloudWatch?

CloudWatch is managed on AWS. Prometheus is free and runs anywhere. Amazon Managed Prometheus and Grafana combine both.

Q. Push or pull?

Prometheus pulls. For short jobs, use the Pushgateway.

Q. How long can Prometheus keep data?

Set by retention. For months or years use Thanos, Mimir or Amazon Managed Prometheus.

Q. Does cAdvisor work on Docker Desktop?

Mostly yes, but some mounts differ. On a Linux VM it is complete.

Q. Can Grafana show logs too?

Yes, with Loki or an Elasticsearch data source from Project 26.

Q. How do I monitor MySQL or Redis?

Add mysqld-exporter or redis-exporter containers and scrape them.

Interview questions

1. How does Prometheus collect metrics?

   Model answer: It scrapes HTTP /metrics endpoints at a fixed interval and stores time series.

2. What is cAdvisor used for?

   Model answer: Per-container resource usage metrics from cgroups.

3. Counter vs gauge vs histogram?

   Model answer: Counter only rises, gauge goes up and down, histogram counts values in buckets for percentiles.

4. What is cardinality and why does it matter?

   Model answer: The number of unique label combinations; too many make Prometheus slow and memory hungry.

5. How do you avoid alert fatigue?

   Model answer: Alert on user impact, use for: durations, group alerts, and route by severity.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview card

मराठी cards