Chapter 58: Docker Projects Part 2: AWS RDS, DynamoDB, S3, Reel Upload, Microservices, Queues, Logging, Monitoring and Cloud Deploys

58.9 Project 25: Kafka event pipeline (Redpanda or Apache Kafka, a producer, consumers and a UI)

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we work at Ola. Every few seconds, each driver's phone sends its location. That is thousands of small events every second. Many teams need these events. The map team shows cars on the map. The pricing team checks demand for surge pricing. The data team stores everything for reports. With a simple job queue (Project 24), one message goes to one worker and then it is gone. But here three teams each need every event. Kafka is built for this. It is an event log. Producers append events to a topic. Each team is a consumer group that reads the whole topic at its own speed. Events stay for days, so a team can even read old events again. We run it in Docker. First with Redpanda, a Kafka-compatible broker in one small container, great for laptops. Then with Apache Kafka in KRaft mode (no ZooKeeper). The Python producer and consumers do not change.

What: every term in plain English

• Event — A small fact that happened, like driver 42 is at this place at this time.
• Topic — A named stream of events, like ride-locations.
• Partition — A topic is split into partitions so many consumers can read in parallel. Order is kept inside one partition.
• Key — Events with the same key (driver id) always go to the same partition, so one driver's events stay in order.
• Producer — An app that writes events to a topic.
• Consumer group — A team of consumers. Each partition is read by only one member of the group.
• Offset — The position of an event in a partition. A group saves the offset it has read up to.
• Retention — How long Kafka keeps events, for example 7 days, even after they are read.
• KRaft — Kafka's built-in controller mode. ZooKeeper is no longer needed.

How: step by step

1. Start a broker. Redpanda in one container plus Redpanda Console UI.
2. Create a topic. ride-locations with 3 partitions.
3. Run the producer. A Python container sends fake driver locations.
4. Run two consumer groups. map and pricing each get every event.
5. Scale a group. 3 consumers in the pricing group split the partitions.
6. Swap to Apache Kafka, then clean up. Same code with the apache/kafka image, then down -v.

Architecture

Folder layout

events/
  compose.yaml         (redpanda, console, producer, map, pricing)
  compose.kafka.yaml   (apache/kafka in KRaft mode)
  .env.kafka
  app/producer.py  app/consumer.py
  app/requirements.txt  app/Dockerfile
  ops/cleanup.sh

Example 1: Redpanda broker and Console with Compose, and creating the topic

Goal. Redpanda broker and Console with Compose, and creating the topic.

compose.yaml

services:
  redpanda:
    image: redpandadata/redpanda:v24.2.7
    command:
      - redpanda
      - start
      - --mode=dev-container
      - --smp=1
      - --memory=1G
      - --kafka-addr=internal://0.0.0.0:9092,external://0.0.0.0:19092
      - --advertise-kafka-addr=internal://redpanda:9092,external://localhost:19092
    ports: ["19092:19092"]
    volumes: [redpanda-data:/var/lib/redpanda/data]
    healthcheck:
      test: ["CMD", "rpk", "cluster", "health", "--exit-when-healthy"]
      interval: 10s
      retries: 10

  console:
    image: redpandadata/console:v2.7.2
    environment:
      KAFKA_BROKERS: redpanda:9092
    ports: ["8080:8080"]
    depends_on:
      redpanda: { condition: service_healthy }

  producer:
    build: ./app
    command: ["python", "producer.py"]
    environment:
      BOOTSTRAP: ${BOOTSTRAP:-redpanda:9092}
      RATE: "20"
    depends_on:
      redpanda: { condition: service_healthy }
    restart: unless-stopped

  map:
    build: ./app
    command: ["python", "consumer.py"]
    environment:
      BOOTSTRAP: ${BOOTSTRAP:-redpanda:9092}
      GROUP: map
    depends_on:
      redpanda: { condition: service_healthy }
    restart: unless-stopped

  pricing:
    build: ./app
    command: ["python", "consumer.py"]
    environment:
      BOOTSTRAP: ${BOOTSTRAP:-redpanda:9092}
      GROUP: pricing
    depends_on:
      redpanda: { condition: service_healthy }
    restart: unless-stopped

volumes:
  redpanda-data:

What each line does

• dev-container mode — Settings made for one small container on a laptop. Use production mode on servers.
• Two listeners — internal redpanda:9092 is for containers, external localhost:19092 is for tools on your laptop.
• advertise-kafka-addr — The address the broker tells clients to use. Wrong values are the most common Kafka error in Docker.
• Console on 8080 — Shows topics, partitions, messages and consumer group lag in the browser.
• BOOTSTRAP default — The apps use Redpanda unless .env says otherwise. That is how we swap to Kafka later.
• Same image, three roles — producer, map and pricing run one image with different commands and env.

Green output

$ docker compose up -d redpanda console
 Container events-redpanda-1  Healthy
 Container events-console-1  Started
$ docker compose exec redpanda rpk topic create ride-locations -p 3 -r 1 -c retention.ms=604800000
TOPIC           STATUS
ride-locations  OK
$ docker compose exec redpanda rpk topic describe ride-locations -p
PARTITION  LEADER  REPLICAS  LOG-START-OFFSET  HIGH-WATERMARK
0          0       [0]       0                 0
1          0       [0]       0                 0
2          0       [0]       0                 0

Expected result. The broker runs and the topic has 3 partitions that keep events for 7 days.

Example 2: Producer and consumer in Python with confluent-kafka

Goal. Producer and consumer in Python with confluent-kafka. The key is the driver id.

app/producer.py

import os, json, time, random, signal
from confluent_kafka import Producer

p = Producer({"bootstrap.servers": os.environ["BOOTSTRAP"], "acks": "all",
              "enable.idempotence": True, "linger.ms": 20})
RATE = int(os.getenv("RATE", "10"))
running = True
signal.signal(signal.SIGTERM, lambda *_: globals().update(running=False))

def done(err, msg):
    if err:
        print("FAILED", err, flush=True)

sent = 0
while running:
    driver = random.randint(1, 50)
    event = {"driver": driver, "lat": round(18.52 + random.uniform(-0.05, 0.05), 5),
             "lon": round(73.85 + random.uniform(-0.05, 0.05), 5), "ts": time.time()}
    p.produce("ride-locations", key=str(driver), value=json.dumps(event), callback=done)
    p.poll(0)
    sent += 1
    if sent % 100 == 0:
        print("sent", sent, flush=True)
    time.sleep(1 / RATE)
p.flush(10)
print("producer stopped after", sent, flush=True)

app/consumer.py

import os, json, socket, signal
from confluent_kafka import Consumer

GROUP = os.environ["GROUP"]
ME = socket.gethostname()
c = Consumer({"bootstrap.servers": os.environ["BOOTSTRAP"], "group.id": GROUP,
              "auto.offset.reset": "earliest", "enable.auto.commit": False})
running = True
signal.signal(signal.SIGTERM, lambda *_: globals().update(running=False))

def assigned(consumer, parts):
    print(ME, GROUP, "got partitions", [x.partition for x in parts], flush=True)
c.subscribe(["ride-locations"], on_assign=assigned)

seen = 0
while running:
    msg = c.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        print("error", msg.error(), flush=True)
        continue
    e = json.loads(msg.value())
    seen += 1
    if GROUP == "pricing" and e["driver"] % 10 == 0:
        print(ME, "pricing: demand check near", e["lat"], e["lon"], flush=True)
    if seen % 100 == 0:
        print(ME, GROUP, "seen", seen, "partition", msg.partition(), "offset", msg.offset(), flush=True)
    c.commit(msg, asynchronous=True)
c.close()

app/requirements.txt

confluent-kafka==2.6.0

app/Dockerfile

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY *.py .
RUN useradd -r app
USER app
CMD ["python", "consumer.py"]

What each line does

• key=str(driver) — All events for driver 7 go to the same partition, so they stay in time order.
• acks=all + idempotence — The broker confirms each write and duplicates from retries are removed.
• linger.ms 20 — The producer waits up to 20 ms to send events in batches. Faster overall.
• auto.offset.reset earliest — A new group starts from the oldest event still kept.
• Manual commit after work — The offset is saved only after we handle the event, so a crash re-reads it instead of losing it.
• on_assign — Prints which partitions this consumer got. We see the split when we scale.

Green output

$ docker compose up -d --build producer map pricing
$ docker compose logs map pricing | grep partitions
map-1      | 6a1b2c3d4e5f map got partitions [0, 1, 2]
pricing-1  | 7b2c3d4e5f6a pricing got partitions [0, 1, 2]
$ docker compose logs --tail 2 map pricing
map-1      | 6a1b2c3d4e5f map seen 400 partition 1 offset 131
pricing-1  | 7b2c3d4e5f6a pricing seen 400 partition 2 offset 140

Expected result. Both groups get every event. One consumer per group reads all 3 partitions.

Example 3: Scale a group, see lag, replay old events, and swap Redpanda for Apache Kafka (KRaft)

Goal. Scale a group, see lag, replay old events, and swap Redpanda for Apache Kafka (KRaft).

compose.kafka.yaml

services:
  kafka:
    image: apache/kafka:3.8.0
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:19093,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:19093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_NUM_PARTITIONS: 3
    ports: ["19093:19093"]
    volumes: [kafka-data:/var/lib/kafka/data]
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list"]
      interval: 10s
      retries: 10

volumes:
  kafka-data:

.env.kafka

BOOTSTRAP=kafka:9092

ops/cleanup.sh

docker compose down -v --rmi local
docker compose -f compose.kafka.yaml down -v 2>/dev/null
echo "events stack removed"

What each line does

• --scale pricing=3 — Three consumers in one group. Kafka gives each one partition. A 4th consumer would sit idle, because there are only 3 partitions.
• Lag — How far a group is behind the newest event. Growing lag means you need more consumers or faster code.
• rpk group seek --to start — Moves a group back to the oldest event. This is replay, impossible with a normal job queue.
• KRaft — PROCESS_ROLES broker,controller: one container does both jobs. No ZooKeeper.
• Two advertised listeners — Same idea as Redpanda: kafka:9092 inside Docker, localhost:19093 outside.
• Only BOOTSTRAP changes — The Python code talks the Kafka protocol, so it works with both brokers.

Green output

$ docker compose up -d --scale pricing=3
$ docker compose logs pricing | grep partitions | tail -3
pricing-1  | ... pricing got partitions [0]
pricing-2  | ... pricing got partitions [1]
pricing-3  | ... pricing got partitions [2]
$ docker compose exec redpanda rpk group describe pricing | head -4
GROUP        pricing
STATE        Stable
MEMBERS      3
TOTAL-LAG    4
$ docker compose stop map && docker compose exec redpanda rpk group seek map --to start && docker compose start map
$ docker compose -f compose.kafka.yaml up -d
$ docker compose -f compose.kafka.yaml exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic ride-locations --partitions 3
Created topic ride-locations.
$ docker compose --env-file .env.kafka up -d --force-recreate producer map pricing
 Container events-producer-1  Started

Expected result. Three consumers split the work, lag is tiny, the map group replayed history, and the same apps run on Apache Kafka.

Every command, step by step (plain docker and Compose)

1. Start the broker

   docker compose up -d redpanda console
   

   What it does: Redpanda and the UI.

   Container events-redpanda-1 Healthy
   

2. Create the topic

   docker compose exec redpanda rpk topic create ride-locations -p 3 -r 1
   

   What it does: 3 partitions, 1 copy (single broker).

   ride-locations OK
   

3. Start apps

   docker compose up -d --build producer map pricing
   

   What it does: Producer sends 20 events per second.

   Container events-producer-1 Started
   

4. Read some events

   docker compose exec redpanda rpk topic consume ride-locations -n 3
   

   What it does: Shows key, value, partition, offset.

   "key": "17"
   

5. Open Console

   http://localhost:8080
   

   What it does: Topics, messages, consumer groups.

   ride-locations 3 partitions
   

6. Check groups

   docker compose exec redpanda rpk group list
   

   What it does: Two groups: map and pricing.

   map Stable / pricing Stable
   

7. Scale pricing

   docker compose up -d --scale pricing=3
   

   What it does: Partitions split over 3 consumers.

   got partitions [1]
   

8. See lag

   docker compose exec redpanda rpk group describe pricing
   

   What it does: TOTAL-LAG stays small.

   TOTAL-LAG 4
   

9. Stop and catch up

   docker compose stop pricing; sleep 30; docker compose start pricing
   

   What it does: Lag grows, then falls as consumers catch up.

   TOTAL-LAG 600 then 3
   

10. Replay

    docker compose stop map && docker compose exec redpanda rpk group seek map --to start && docker compose start map
    

    What it does: map reads all old events again.

    seen 100 ... offset 33
    

11. Swap to Kafka

    docker compose -f compose.kafka.yaml up -d
    

    What it does: Apache Kafka in KRaft mode on the same project network.

    Container events-kafka-1 Started
    

12. Clean up

    bash ops/cleanup.sh
    

    What it does: Removes containers, volumes and images.

    events stack removed
    

When it is red 1

KafkaError{code=_TRANSPORT,val=-195,str="localhost:9092/0: Connect to ipv4#127.0.0.1:9092 failed: Connection refused"}

Cause. The app is inside Docker but the broker advertised localhost.

Fix, step by step

1. Inside containers use redpanda:9092 or kafka:9092.
2. Check advertise-kafka-addr has an internal listener with the service name.

When it is red 2

Subscribed topic not available: ride-locations: Broker: Unknown topic or partition

Cause. The topic was not created and auto create is off.

Fix, step by step

1. Create it with rpk topic create or kafka-topics.sh --create.

When it is red 3

One consumer says got partitions []

Cause. The group has more consumers than partitions.

Fix, step by step

1. Partitions limit parallelism. Add partitions with rpk topic add-partitions, or run fewer consumers.

When it is red 4

Consumers read the same events again after every restart

Cause. Offsets are never committed, or commit happens before the group id is set.

Fix, step by step

1. Call commit after handling each event, or enable auto commit.

When it is red 5

redpanda exited with code 1: Memory: 1G is greater than available

Cause. Docker Desktop has too little RAM.

Fix, step by step

1. Give Docker Desktop 4 GB or more, or lower --memory to 512M.

When it is red 6

Events for one driver arrive out of order

Cause. Events were produced without a key, so they spread over partitions.

Fix, step by step

1. Always set key to the driver id.

Practice tasks

docker compose up -d redpanda console

rpk topic describe ride-locations -p

rpk topic consume ride-locations -n 5

rpk group describe map

--scale pricing=4

rpk group describe pricing

rpk group seek map --to start

BOOTSTRAP=kafka:9092

GROUP=archive

bash ops/cleanup.sh

Recap

• Kafka is an event log: producers append, consumer groups read at their own pace.
• Every group gets every event; inside a group, partitions are split between consumers.
• Use a key to keep related events in order.
• Commit offsets after handling an event; replay by seeking a group.
• Advertised listeners must match where the client runs: service name inside Docker, localhost outside.
• Redpanda is light for laptops; Apache Kafka in KRaft mode needs no ZooKeeper.

FAQ

Q. Kafka or RabbitMQ?

RabbitMQ for task queues and routing. Kafka for high-volume event streams that many teams read and replay.

Q. Is Redpanda really Kafka?

It speaks the Kafka protocol, so Kafka clients work. It is a different program written in C++.

Q. How many partitions should I create?

Enough for the most consumers you will run in one group. You can add partitions later, but key order can change.

Q. What about production on AWS?

Amazon MSK runs Kafka for you. Only the bootstrap address and security settings change.

Q. How long are events kept?

As long as retention says, by time or size, read or not.

Q. Do I need Kafka Connect?

Only to move data to and from databases and S3 without code. Run it as another container.

Interview questions

1. What is a Kafka consumer group?

   Model answer: A set of consumers sharing a topic's partitions; each partition goes to one member.

2. How does Kafka keep order?

   Model answer: Only inside a partition; use a key to send related events to one partition.

3. What is an offset and who stores it?

   Model answer: The position in a partition; the group commits it to Kafka.

4. What happens if a consumer crashes?

   Model answer: The group rebalances and another member takes its partitions from the last committed offset.

5. Why do advertised listeners matter in Docker?

   Model answer: Clients connect to the address the broker advertises, so it must be reachable from where the client runs.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview card

मराठी cards