Chapter 83: DevOps Projects Part 13: Monitoring Prometheus Grafana Alertmanager EFK Loki CloudWatch OpenTelemetry

83.4 Project 94: Loki + Grafana for Kubernetes logs

By Ravindra Bagale Last updated: 28 October 2026

Friends, full ELK is heavy for many teams. Today we run Loki, ship with Promtail, and query logs next to metrics in Grafana.

What you will build in this project

Run Loki.
Ship with Promtail.
Add Loki datasource.
Query logs beside a metric panel.

1. Start from a real company story

At Flipkart in Bengaluru, metrics showed errors but log search lived in another tool and slowed the war room.

Rough cost: about Rs 12 lakh in extended incident time.

Aditi put Loki beside Prometheus in the same Grafana.

Today we query both in one place.

Picture 1: pods -> Promtail -> Loki -> Grafana Explore

Ravindra Bagale's Tip

Students use high-cardinality labels like request_id on Loki streams. Keep labels low; put ids in the line. Keep that habit.

2. Start and prepare

Step 1: Why this project

Why: Real outages teach. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: echo why. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 1: a1

Type this:

echo 'Today we build loki94'

You will see:

Today we build loki94

What it means: Why.

Example 2: a2

Type this:

mkdir -p /tmp/loki94 && echo 'Scope ready'

You will see:

Scope ready

What it means: Scope.

Example 3: a3

Type this:

echo 'Keep commands short and check every output'

You will see:

Keep commands short and check every output

What it means: Habit.

Step 2: Prepare workspace

Why: Folders and notes. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: prep. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 4: b1

Type this:

mkdir -p /tmp/loki94 && printf '%s\n' 'lab notes' > /tmp/loki94/NOTES.txt && cat /tmp/loki94/NOTES.txt

You will see:

lab notes

What it means: Notes.

Example 5: b2

Type this:

echo 'Work only under /tmp/loki94 for this project'

You will see:

Work only under /tmp/loki94 for this project

What it means: Path.

Example 6: b3

Type this:

echo 'Write ACCEPT.txt at the end'

You will see:

Write ACCEPT.txt at the end

What it means: Accept file.

Step 3: Loki config

Why: Single binary. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: loki. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 7: c1

Type this:

echo 'docker run -d --name loki -p 3100:3100 grafana/loki:2.9.4 -config.file=/etc/loki/local-config.yaml'

You will see:

docker run -d --name loki -p 3100:3100 grafana/loki:2.9.4 -config.file=/etc/loki/local-config.yaml

What it means: Loki.

Example 8: c2

Type this:

echo 'curl -s http://localhost:3100/ready'

You will see:

curl -s http://localhost:3100/ready

What it means: Ready.

Example 9: c3

Type this:

echo 'Lab uses filesystem store; prod often uses object storage'

You will see:

Lab uses filesystem store; prod often uses object storage

What it means: Store.

3. Build the main pieces

Step 4: Promtail

Why: DaemonSet shipper. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: promtail. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 10: d1

Type this:

printf '%s\n' 'clients:' '- url: http://loki:3100/loki/api/v1/push' 'scrape_configs:' '- job_name: kubernetes-pods' '  kubernetes_sd_configs:' '  - role: pod' > /tmp/loki94/promtail.yml && cat /tmp/loki94/promtail.yml

You will see:

kubernetes_sd_configs:

What it means: Promtail.

Example 11: d2

Type this:

echo 'Run Promtail as a DaemonSet with access to container logs'

You will see:

Run Promtail as a DaemonSet with access to container logs

What it means: DS.

Example 12: d3

Type this:

echo 'Drop noisy debug lines at the shipper when possible'

You will see:

Drop noisy debug lines at the shipper when possible

What it means: Drop.

Step 5: Grafana Loki DS

Why: Explore. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: ds. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 13: e1

Type this:

printf '%s\n' 'Add datasource Loki URL http://loki:3100' 'Explore: {app="shop-api"} |= "error"' > /tmp/loki94/DS.txt && cat /tmp/loki94/DS.txt

You will see:

Explore: {app="shop-api"} |= "error"

What it means: DS.

Example 14: e2

Type this:

echo 'LogQL uses labels then line filters'

You will see:

LogQL uses labels then line filters

What it means: LogQL.

Example 15: e3

Type this:

printf '%s\n' 'Acceptance: Explore shows error lines for shop-api in the last 15m' > /tmp/loki94/ACCEPT.txt && cat /tmp/loki94/ACCEPT.txt

You will see:

Acceptance: Explore shows error lines for shop-api in the last 15m

What it means: Gate.

Step 6: Label rules

Why: Cardinality. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: labels. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 16: f1

Type this:

printf '%s\n' 'Good labels: namespace, app, level' 'Bad labels: request_id, user_id, full path' > /tmp/loki94/LABELS.txt && cat /tmp/loki94/LABELS.txt

You will see:

Bad labels: request_id, user_id, full path

What it means: Labels.

Example 17: f2

Type this:

echo 'Put high-cardinality ids in the log line body'

You will see:

Put high-cardinality ids in the log line body

What it means: Body.

Example 18: f3

Type this:

echo 'Too many streams will burn Loki memory'

You will see:

Too many streams will burn Loki memory

What it means: Risk.

4. Automate and verify

Step 7: Metrics to logs

Why: Derived fields. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: link. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 19: g1

Type this:

echo 'In Grafana, link a Prometheus panel to Loki Explore with labels'

You will see:

In Grafana, link a Prometheus panel to Loki Explore with labels

What it means: Link.

Example 20: g2

Type this:

echo 'Same labels on metrics and logs make jumps easy'

You will see:

Same labels on metrics and logs make jumps easy

What it means: Align.

Example 21: g3

Type this:

echo 'War rooms move faster when the jump is one click'

You will see:

War rooms move faster when the jump is one click

What it means: Speed.

Step 8: Retention

Why: Object storage. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: retain. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 22: h1

Type this:

printf '%s\n' 'Set retention_period in Loki limits' 'Prod: S3/GCS chunks with compaction' > /tmp/loki94/RET.txt && cat /tmp/loki94/RET.txt

You will see:

Set retention_period in Loki limits

What it means: Retain.

Example 23: h2

Type this:

echo '14 days is a common app-log default'

You will see:

14 days is a common app-log default

What it means: Default.

Example 24: h3

Type this:

echo 'Compliance logs may need a separate longer store'

You will see:

Compliance logs may need a separate longer store

What it means: Compliance.

Step 9: Compare to EFK

Why: When Loki. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: compare. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 25: i1

Type this:

printf '%s\n' 'Choose Loki when Grafana-centric and cost-sensitive' 'Choose EFK when you need heavy ad-hoc analytics on logs' > /tmp/loki94/CMP.txt && cat /tmp/loki94/CMP.txt

You will see:

Choose Loki when Grafana-centric and cost-sensitive

What it means: Compare.

Example 26: i2

Type this:

echo 'Either way: structured logs and request ids'

You will see:

Either way: structured logs and request ids

What it means: Shared.

Example 27: i3

Type this:

echo 'Do not run both full stacks for the same logs without a reason'

You will see:

Do not run both full stacks for the same logs without a reason

What it means: One.

5. Harden and accept

Step 10: Story Flipkart

Why: Bengaluru lesson. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: story. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 28: j1

Type this:

printf '%s\n' 'At Flipkart in Bengaluru, metrics and logs lived in different tools in the war room.' 'Aditi put Loki beside Prometheus in Grafana.' > /tmp/loki94/STORY.txt && wc -l /tmp/loki94/STORY.txt

You will see:

2 /tmp/loki94/STORY.txt

What it means: Story.

Example 29: j2

Type this:

echo 'Rough cost: about Rs 12 lakh in extended incident time.'

You will see:

Rough cost: about Rs 12 lakh in extended incident time.

What it means: Cost.

Example 30: j3

Type this:

echo 'Aditi: PASS'

You will see:

Aditi: PASS

What it means: Sign-off.

Step 11: Proof

Why: Files. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: ls. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 31: k1

Type this:

ls /tmp/loki94

You will see:

NOTES.txt\nSTORY.txt\nACCEPT.txt

What it means: Tree.

Example 32: k2

Type this:

test -f /tmp/loki94/ACCEPT.txt && echo ACCEPT_OK

You will see:

ACCEPT_OK

What it means: Accept present.

Example 33: k3

Type this:

echo 'Next: CloudWatch dashboards project 95'

You will see:

Next: CloudWatch dashboards project 95

What it means: Bridge.

Step 12: Acceptance

Why: Loki Explore works. We do this before the next step so mistakes stay small and easy to undo. Read the output carefully; the numbers and names tell you if the machine is healthy.

What: final. Follow the numbered examples below. Type each command, read the exact output, and match the meaning line before you move on. That is how real teams learn a tool.

Example 34: z1

Type this:

echo 'Loki ready'

You will see:

Loki ready

What it means: A.

Example 35: z2

Type this:

echo 'Promtail shipping'

You will see:

Promtail shipping

What it means: B.

Example 36: z3

Type this:

echo 'Explore query hits'

You will see:

Explore query hits

What it means: C.

Check yourself

Q1. Who fixed Flipkart war room?

Aditi

Q2. Log store?

Loki

Q3. Shipper?

Promtail

Q4. UI?

Grafana

Q5. request_id as label?

Q6. Beside metrics?

yes

Friends, you queried logs next to metrics. Did you understand? Let us try it once more. Spike an error metric and jump to matching Loki lines in Explore.

Lab: try at home

Loki up.
Promtail shipping.
Grafana Loki DS.
Explore jump works.

Recap

Loki is log aggregation.
Low cardinality labels.
Promtail DaemonSet.
Same Grafana.
Link metrics to logs.