Chapter 58: Docker Projects Part 2: AWS RDS, DynamoDB, S3, Reel Upload, Microservices, Queues, Logging, Monitoring and Cloud Deploys

58.10 Project 26: ELK and EFK logging for the microservices shop (Elasticsearch, Fluent Bit or Logstash, Kibana)

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Remember the microservices shop from Project 21: a gateway, users in Node.js, orders in Python and payments in Go. One evening a customer at Myntra-style scale says "my payment failed". Which container has the error? With docker logs we must open five containers, one by one, and scroll. If a container was recreated, its old logs are gone. In real companies all logs go to one central place where we can search: "show every ERROR from payments in the last 15 minutes for order 4521". The classic stack is ELK: Elasticsearch stores and searches logs, Logstash processes them, Kibana shows them. Many teams swap Logstash for Fluent Bit, a tiny log shipper, and call it EFK. In this project we run EFK with Docker. Fluent Bit collects logs from every shop container through the fluentd logging driver, adds the service name, and sends them to Elasticsearch. Then we search in Kibana. At the end we see the Logstash version too.

What: every term in plain English

• Centralised logging — All containers send logs to one store, so we search in one place.
• Elasticsearch — A search database. It stores each log line as a JSON document in an index.
• Index — Like a table in Elasticsearch. We use one per day, such as shop-2026.10.25.
• Kibana — The web UI for Elasticsearch: Discover to search, dashboards to chart.
• Fluent Bit — A very small log collector and shipper written in C. Uses a few MB of RAM.
• Logstash — A bigger log pipeline in Java with many filters. The L in ELK.
• fluentd logging driver — A Docker logging driver that sends each container's stdout to Fluent Bit or Fluentd on port 24224.
• Structured log — A log line in JSON, so fields like level and order_id can be searched directly.

How: step by step

1. Start Elasticsearch and Kibana. Single node, security on with a placeholder password.
2. Start Fluent Bit. Listens on 24224 and sends to Elasticsearch.
3. Point the shop at Fluent Bit. Add the fluentd logging driver to each service with a tag.
4. Search in Kibana. Create a data view shop-* and filter by service and level.
5. Try Logstash. Swap Fluent Bit for Logstash with a grok filter.
6. Clean up. down -v removes the stack and indexes.

Architecture

Folder layout

project/
  shop/                       (from Project 21)
    compose.yaml
    compose.logging.yaml      (adds the fluentd driver)
  logging/
    compose.yaml              (elasticsearch, kibana-setup, kibana, fluent-bit)
    compose.logstash.yaml     (ELK version)
    .env.example
    fluent-bit/fluent-bit.conf  fluent-bit/parsers.conf
    logstash/pipeline.conf
    ilm-policy.sh  cleanup.sh

Example 1: Elasticsearch, Kibana and Fluent Bit in one compose file

Goal. Elasticsearch, Kibana and Fluent Bit in one compose file.

logging/compose.yaml

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.3
    environment:
      discovery.type: single-node
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
      xpack.security.enabled: "true"
      xpack.security.http.ssl.enabled: "false"
      ES_JAVA_OPTS: -Xms1g -Xmx1g
    ulimits:
      memlock: { soft: -1, hard: -1 }
    volumes: [es-data:/usr/share/elasticsearch/data]
    ports: ["127.0.0.1:9200:9200"]
    healthcheck:
      test: ["CMD-SHELL", "curl -s -u elastic:${ELASTIC_PASSWORD} localhost:9200/_cluster/health | grep -qE 'green|yellow'"]
      interval: 10s
      retries: 30

  kibana-setup:
    image: curlimages/curl:8.10.1
    depends_on: { elasticsearch: { condition: service_healthy } }
    command: ["sh", "-c", "curl -s -u elastic:${ELASTIC_PASSWORD} -X POST -H 'Content-Type: application/json' http://elasticsearch:9200/_security/user/kibana_system/_password -d '{\"password\":\"${KIBANA_PASSWORD}\"}' && echo kibana password set"]
    restart: "no"

  kibana:
    image: docker.elastic.co/kibana/kibana:8.15.3
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: ${KIBANA_PASSWORD}
    ports: ["127.0.0.1:5601:5601"]
    depends_on:
      kibana-setup: { condition: service_completed_successfully }

  fluent-bit:
    image: fluent/fluent-bit:3.1
    environment:
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
    volumes:
      - ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./fluent-bit/parsers.conf:/fluent-bit/etc/parsers.conf:ro
    ports: ["127.0.0.1:24224:24224"]
    depends_on: { elasticsearch: { condition: service_healthy } }
    restart: unless-stopped

volumes:
  es-data:

logging/.env.example

ELASTIC_PASSWORD=change-me-elastic
KIBANA_PASSWORD=change-me-kibana

What each line does

• single-node — One Elasticsearch node for learning. Production uses 3 or more nodes, or Amazon OpenSearch Service.
• ES_JAVA_OPTS 1g — Elasticsearch is Java. We fix the heap at 1 GB so it does not take all RAM.
• 127.0.0.1 ports — Elasticsearch, Kibana and Fluent Bit are reachable only from this machine, not from the internet.
• kibana-setup — A one-time job that sets the kibana_system password. Kibana starts only after it succeeds.
• security on — Elasticsearch 8 has login on by default. We keep it and use placeholder passwords from .env.
• 24224 — The port the Docker fluentd driver sends logs to.

Green output

$ cd logging && cp .env.example .env
$ docker compose up -d
 Container logging-elasticsearch-1  Healthy
 Container logging-kibana-setup-1  Exited (0)
 Container logging-kibana-1  Started
 Container logging-fluent-bit-1  Started
$ curl -s -u elastic:change-me-elastic localhost:9200 | grep number
    "number" : "8.15.3",

Expected result. Storage, UI and shipper are running. Next we tell Fluent Bit what to do with logs.

Example 2: Fluent Bit config and the shop services sending logs with the fluentd driver

Goal. Fluent Bit config and the shop services sending logs with the fluentd driver.

logging/fluent-bit/fluent-bit.conf

[SERVICE]
    Flush        2
    Log_Level    info
    Parsers_File parsers.conf
    HTTP_Server  On
    HTTP_Port    2020

[INPUT]
    Name   forward
    Listen 0.0.0.0
    Port   24224

[FILTER]
    Name         parser
    Match        shop.*
    Key_Name     log
    Parser       json_safe
    Reserve_Data On

[FILTER]
    Name   record_modifier
    Match  shop.*
    Record env dev

[OUTPUT]
    Name               es
    Match              shop.*
    Host               elasticsearch
    Port               9200
    HTTP_User          elastic
    HTTP_Passwd        ${ELASTIC_PASSWORD}
    Logstash_Format    On
    Logstash_Prefix    shop
    Suppress_Type_Name On
    Replace_Dots       On
    Retry_Limit        5

logging/fluent-bit/parsers.conf

[PARSER]
    Name        json_safe
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ
    Time_Keep   On

shop/compose.logging.yaml

x-log: &log
  driver: fluentd
  options:
    fluentd-address: 127.0.0.1:24224
    fluentd-async: "true"
    tag: shop.{{.Name}}

services:
  gateway:
    logging: *log
  users:
    logging: *log
  orders:
    logging: *log
  payments:
    logging: *log

shop/payments/log_example.go

// Structured JSON log line from the payments service (Go, log/slog)
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
logger.Error("payment failed", "order_id", 4521, "amount", 1299, "reason", "card declined", "service", "payments")

What each line does

• forward input — Receives logs from the Docker fluentd driver.
• parser json_safe — If the app writes JSON, its fields (level, order_id) become separate searchable fields.
• Reserve_Data On — Keeps container_name and other fields the driver adds.
• Logstash_Format + Prefix shop — Writes to daily indexes shop-2026.10.25. Old days are easy to delete.
• tag shop.{{.Name}} — Each line is tagged with the container name, so Match shop.* catches the whole shop.
• fluentd-address 127.0.0.1 — The driver runs in the Docker engine on the host, not in a container, so it uses the host port.
• fluentd-async true — If Fluent Bit is down, containers still start. Without it, a container fails to start.
• compose.logging.yaml — An override file: the shop code does not change, we only add logging.

Green output

$ cd ../shop
$ docker compose -f compose.yaml -f compose.logging.yaml up -d
$ curl -s -X POST localhost:8080/api/orders -H 'Content-Type: application/json' -d '{"user":1,"item":"shoes","card":"4000000000000002"}'
{"error":"payment failed"}
$ curl -s -u elastic:change-me-elastic 'localhost:9200/_cat/indices/shop-*?v&h=index,docs.count'
index             docs.count
shop-2026.10.25   184
$ curl -s -u elastic:change-me-elastic 'localhost:9200/shop-*/_search?q=order_id:4521&size=1' | grep -o '"reason":"[a-z ]*"'
"reason":"card declined"

Expected result. Logs from all shop containers are in one index, and we found the payment error by order id in one query.

Example 3: Search in Kibana, add retention, and the Logstash (ELK) version of the pipeline

Goal. Search in Kibana, add retention, and the Logstash (ELK) version of the pipeline.

logging/ilm-policy.sh

U=elastic:change-me-elastic
curl -s -u "$U" -X PUT localhost:9200/_ilm/policy/shop-7d -H 'Content-Type: application/json' -d '{"policy":{"phases":{"hot":{"actions":{}},"delete":{"min_age":"7d","actions":{"delete":{}}}}}}'
curl -s -u "$U" -X PUT localhost:9200/_index_template/shop -H 'Content-Type: application/json' -d '{"index_patterns":["shop-*"],"template":{"settings":{"index.lifecycle.name":"shop-7d","number_of_replicas":0}}}'
echo

logging/logstash/pipeline.conf

input {
  beats { port => 5044 }
  tcp { port => 5000 codec => json_lines }
}
filter {
  if [message] =~ /^\{/ {
    json { source => "message" }
  } else {
    grok { match => { "message" => "%{IPORHOST:client} - - \[%{HTTPDATE:ts}\] \"%{WORD:method} %{URIPATH:path}[^\"]*\" %{NUMBER:status:int} %{NUMBER:bytes:int}" } }
  }
  mutate { add_field => { "env" => "dev" } }
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    user => "elastic"
    password => "${ELASTIC_PASSWORD}"
    index => "shop-logstash-%{+YYYY.MM.dd}"
  }
}

logging/compose.logstash.yaml

services:
  logstash:
    image: docker.elastic.co/logstash/logstash:8.15.3
    environment:
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
      LS_JAVA_OPTS: -Xms512m -Xmx512m
    volumes: [./logstash/pipeline.conf:/usr/share/logstash/pipeline/logstash.conf:ro]
    ports: ["127.0.0.1:5000:5000"]
    depends_on: { elasticsearch: { condition: service_healthy } }

logging/cleanup.sh

cd "$(dirname "$0")"
(cd ../shop && docker compose -f compose.yaml -f compose.logging.yaml down)
docker compose -f compose.yaml -f compose.logstash.yaml down -v
echo "logging stack and indexes removed"

What each line does

• ILM policy shop-7d — Index lifecycle management deletes shop indexes older than 7 days. Disks do not fill up.
• number_of_replicas 0 — A single node cannot hold replicas; without this the cluster stays yellow.
• Kibana data view shop-* — In Stack Management, Data Views, create shop-* with @timestamp. Then use Discover.
• KQL search — In Discover type: container_name : payments and level : ERROR and order_id : 4521
• Logstash json and grok — JSON lines are parsed directly; nginx access lines are cut into fields with a grok pattern.
• Logstash is heavier — It needs about 512 MB to 1 GB of RAM. Fluent Bit needs about 10 MB. Many teams use Fluent Bit on hosts and Logstash centrally.

Green output

$ bash ilm-policy.sh
{"acknowledged":true}{"acknowledged":true}
$ docker compose -f compose.yaml -f compose.logstash.yaml up -d logstash
$ echo '{"service":"orders","level":"ERROR","msg":"db timeout","order_id":4522}' | nc -q1 localhost 5000
$ curl -s -u elastic:change-me-elastic 'localhost:9200/_cat/indices/shop-*?h=index'
shop-2026.10.25
shop-logstash-2026.10.25
$ bash cleanup.sh
logging stack and indexes removed

Expected result. Old logs now expire after 7 days, and the same search works with a Logstash pipeline.

Every command, step by step (plain docker and Compose)

1. Raise the mmap limit (Linux)

   sudo sysctl -w vm.max_map_count=262144
   

   What it does: Elasticsearch needs this on Linux hosts. Docker Desktop already has it.

   vm.max_map_count = 262144
   

2. Copy env

   cd logging && cp .env.example .env
   

   What it does: Placeholder passwords; change them.

   (no output)
   

3. Start EFK

   docker compose up -d
   

   What it does: Elasticsearch, Kibana and Fluent Bit.

   Container logging-kibana-1 Started
   

4. Check health

   curl -s -u elastic:change-me-elastic localhost:9200/_cluster/health?pretty | grep status
   

   What it does: Green or yellow is fine for one node.

   "status" : "green"
   

5. Check Fluent Bit

   curl -s localhost:2020/api/v1/metrics 2>/dev/null || docker compose logs fluent-bit | tail -3
   

   What it does: Fluent Bit is up and connected.

   [output:es:es.0] worker #0 started
   

6. Start the shop with logging

   cd ../shop && docker compose -f compose.yaml -f compose.logging.yaml up -d
   

   What it does: Every service now logs to Fluent Bit.

   Container shop-payments-1 Started
   

7. Make traffic

   for i in $(seq 1 20); do curl -s localhost:8080/api/users/1 -o /dev/null; done
   

   What it does: Creates log lines.

   (no output)
   

8. List indexes

   curl -s -u elastic:change-me-elastic 'localhost:9200/_cat/indices/shop-*?v'
   

   What it does: One daily index with documents.

   shop-2026.10.25 ... 184
   

9. Open Kibana

   http://localhost:5601 (elastic / change-me-elastic)
   

   What it does: Create data view shop-* and open Discover.

   Discover shows hits
   

10. Search errors

    In Discover: level : ERROR and container_name : *payments*
    

    What it does: Only payment errors.

    1 hit: card declined
    

11. Note docker logs

    docker logs shop-payments-1
    

    What it does: With the fluentd driver plus Docker 20.10+ dual logging, docker logs still works locally.

    payment failed ...
    

12. Clean up

    bash ../logging/cleanup.sh
    

    What it does: Removes stacks and the es-data volume.

    logging stack and indexes removed
    

When it is red 1

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Cause. Linux host setting is too low for Elasticsearch.

Fix, step by step

1. sudo sysctl -w vm.max_map_count=262144
2. Add it to /etc/sysctl.conf to keep it after reboot.

When it is red 2

Error response from daemon: failed to initialize logging driver: dial tcp 127.0.0.1:24224: connect: connection refused

Cause. The container uses the fluentd driver but Fluent Bit is not running, and async is off.

Fix, step by step

1. Start the logging stack first.
2. Set fluentd-async: true.

When it is red 3

[output:es:es.0] HTTP status=401 URI=/_bulk

Cause. Wrong Elasticsearch user or password in Fluent Bit.

Fix, step by step

1. Check ELASTIC_PASSWORD reaches the fluent-bit container.
2. Test with curl -u elastic:... localhost:9200.

When it is red 4

[output:es:es.0] error: Action/metadata line [1] contains an unknown parameter [_type]

Cause. Elasticsearch 8 removed types.

Fix, step by step

1. Set Suppress_Type_Name On in the es output.

When it is red 5

Kibana server is not ready yet

Cause. Kibana is still starting or cannot log in with kibana_system.

Fix, step by step

1. Wait 1 to 2 minutes.
2. Check kibana-setup exited with 0 and the password matches.

When it is red 6

elasticsearch exited with code 137

Cause. Out of memory. Elasticsearch plus Kibana need around 3 GB.

Fix, step by step

1. Give Docker Desktop 6 GB, or lower ES_JAVA_OPTS to 512m for practice.

Practice tasks

docker compose up -d

_cat/indices/shop-*?v

container_name : *users*

order_id : 4521

GET shop-*/_ilm/explain

fluentd-async

nc localhost 5000

Visualize Library

Stack Management, Rules

bash logging/cleanup.sh

Recap

• Central logging puts every container's logs in one searchable place.
• EFK: Fluent Bit ships, Elasticsearch stores, Kibana searches; ELK uses Logstash instead.
• The Docker fluentd driver plus a tag per container is the simplest way to ship logs.
• Write JSON logs so fields are searchable.
• Keep Elasticsearch private, give it enough RAM, and delete old indexes with ILM.
• Use an override compose file so the app code does not change.

FAQ

Q. Can I use Loki instead of Elasticsearch?

Yes. Grafana Loki is lighter and pairs with Grafana. Project 27 uses Grafana for metrics.

Q. What about AWS?

Amazon OpenSearch Service replaces Elasticsearch, or send logs to CloudWatch Logs with the awslogs driver.

Q. Does docker logs still work?

Yes on Docker 20.10 and newer with dual logging, for local debugging.

Q. Why not mount /var/lib/docker/containers instead?

That also works with a tail input. The driver is easier to start with.

Q. How much disk do logs need?

Measure documents per day times average size. Delete old indexes with ILM.

Q. Is OpenSearch the same as Elasticsearch?

It started as a fork of Elasticsearch 7. Most ideas and APIs are the same.

Interview questions

1. How do you collect logs from many containers?

   Model answer: A logging driver or a host agent such as Fluent Bit ships them to a central store.

2. What is the difference between ELK and EFK?

   Model answer: The shipper: Logstash in ELK, Fluent Bit or Fluentd in EFK.

3. Why use structured logging?

   Model answer: JSON fields can be filtered and charted without fragile text parsing.

4. How do you control Elasticsearch storage?

   Model answer: Daily indexes, ILM delete phase, and fewer replicas on small clusters.

5. What happens if the log shipper is down?

   Model answer: With async the app keeps running and logs buffer; without it containers may fail to start.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview cardComparison card

मराठी cards