Chapter 58: Docker Projects Part 2: AWS RDS, DynamoDB, S3, Reel Upload, Microservices, Queues, Logging, Monitoring and Cloud Deploys

58.5 Project 21: Microservices shop (nginx gateway, users in Node, orders in Python, payments in Go, Postgres and MongoDB)

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we work at Flipkart in its early days. One big app does everything: login, cart, orders, payments. Every small change needs a full deploy, and one bug in payments can take down login too. So the teams split the app into small services. Each service does one job, has its own database, and can be deployed alone. The users team writes Node.js with MongoDB. The orders team writes Python with Postgres. The payments team writes Go and keeps no database at all in this demo. The customer should not know about three services. They see one address. An nginx gateway in front sends /api/users to users, /api/orders to orders, and /api/payments to payments. Services talk to each other by name over a private Docker network. Orders calls users to check the customer, and calls payments to charge. We add health checks, timeouts, a request id that travels through every service, and separate networks so the databases are not reachable from the gateway. Then we update one service alone without touching the others. That is the real power of microservices.

What: every term in plain English

• Microservice — A small app that owns one business job and its own data, and is deployed by itself.
• API gateway — One front door. It routes each path to the right service and hides the inside from users.
• Database per service — Orders cannot read the users database directly. It must ask the users service by API.
• Service discovery — In Docker, each Compose service name is a DNS name, such as http://users:3000.
• Request id — An X-Request-Id header created at the gateway and passed on, so we can follow one request in every log.
• Timeout — How long one service waits for another. Without it, one slow service freezes all the others.
• Internal network — A Docker network with internal: true. Containers on it cannot reach the internet, and the gateway is not on it.
• Independent deploy — docker compose up -d --no-deps --build orders rebuilds and restarts only orders.

How: step by step

1. Plan the services and networks. edge network for the gateway, backend for services, data networks per database.
2. Write the three services. Each has /healthz, its own Dockerfile and its own database.
3. Write the gateway config. nginx routes by path and adds X-Request-Id.
4. Start everything with Compose. depends_on with health conditions.
5. Place an order end to end. gateway to orders to users and payments.
6. Break and update one service. Stop payments, see a clean error, then deploy a new payments version alone.
7. Clean up. down -v.

Architecture

Folder layout

shop/
  compose.yaml
  gateway/nginx.conf
  users/index.js  users/package.json  users/Dockerfile
  orders/app.py  orders/Dockerfile
  payments/main.go  payments/go.mod  payments/Dockerfile
  ops/order-flow.sh  ops/failure.sh
  ops/deploy-payments-v2.sh  ops/cleanup.sh

Example 1: The plan in one compose

Goal. The plan in one compose.yaml: four app containers, two databases, three networks, health checks everywhere.

compose.yaml

services:
  gateway:
    image: nginx:1.27-alpine
    volumes:
      - ./gateway/nginx.conf:/etc/nginx/conf.d/default.conf:ro
    ports: ["8080:80"]
    networks: [edge, backend]
    depends_on:
      users: { condition: service_healthy }
      orders: { condition: service_healthy }
      payments: { condition: service_healthy }

  users:
    build: ./users
    environment:
      MONGO_URL: mongodb://users-db:27017/users
    networks: [backend, users-data]
    depends_on: { users-db: { condition: service_healthy } }
    healthcheck: { test: ["CMD", "wget", "-qO-", "http://localhost:3000/healthz"], interval: 10s, retries: 5 }

  orders:
    build: ./orders
    environment:
      DATABASE_URL: postgresql://orders:change-me-orders@orders-db:5432/orders
      USERS_URL: http://users:3000
      PAYMENTS_URL: http://payments:8080
    networks: [backend, orders-data]
    depends_on: { orders-db: { condition: service_healthy } }
    healthcheck: { test: ["CMD", "python", "-c", "import urllib.request as u; u.urlopen('http://localhost:8000/healthz')"], interval: 10s, retries: 5 }

  payments:
    build: ./payments
    environment:
      FAIL_ABOVE: "50000"
    networks: [backend]
    healthcheck: { test: ["CMD", "/payments", "-health"], interval: 10s, retries: 5 }

  users-db:
    image: mongo:7
    volumes: [users-data:/data/db]
    networks: [users-data]
    healthcheck: { test: ["CMD", "mongosh", "--quiet", "--eval", "db.runCommand({ping:1}).ok"], interval: 10s, retries: 10 }

  orders-db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: orders
      POSTGRES_PASSWORD: change-me-orders
      POSTGRES_DB: orders
    volumes: [orders-data:/var/lib/postgresql/data]
    networks: [orders-data]
    healthcheck: { test: ["CMD", "pg_isready", "-U", "orders"], interval: 5s, retries: 10 }

networks:
  edge:
  backend:
  users-data: { internal: true }
  orders-data: { internal: true }

volumes:
  users-data:
  orders-data:

What each line does

• Only gateway has ports — Users reach only port 8080. Services and databases have no published ports.
• networks: [backend, users-data] — users can reach its own Mongo and other services, but not the orders Postgres.
• internal: true — Data networks have no internet route. A hacked database container cannot call out.
• Gateway not on data networks — Even nginx cannot reach a database directly.
• depends_on service_healthy — The gateway starts only when all three services answer their health checks.
• /payments -health — The Go binary has a small health flag, because the distroless image has no curl or wget.
• FAIL_ABOVE — A demo rule: payments above 50000 rupees are declined, so we can test the failure path.

Green output

$ docker compose config --services
gateway
users
orders
payments
users-db
orders-db
$ docker compose config --format json | python3 -c "import json,sys; d=json.load(sys.stdin); print({k: list(v['networks']) for k,v in d['services'].items()})"
{'gateway': ['backend', 'edge'], 'orders': ['backend', 'orders-data'], 'orders-db': ['orders-data'], 'payments': ['backend'], 'users': ['backend', 'users-data'], 'users-db': ['users-data']}

Expected result. The design is clear before any code: one door, private services, and each database on its own internal network.

Example 2: The three services: users (Node + MongoDB), orders (Python + Postgres, calls the others with timeouts), payments (Go, distroless)

Goal. The three services: users (Node + MongoDB), orders (Python + Postgres, calls the others with timeouts), payments (Go, distroless).

users/index.js

const express = require("express");
const { MongoClient, ObjectId } = require("mongodb");
const client = new MongoClient(process.env.MONGO_URL);
const app = express();
app.use(express.json());
let users;

app.use((req, res, next) => {
  console.log(JSON.stringify({ svc: "users", rid: req.get("x-request-id"), path: req.path }));
  next();
});
app.get("/healthz", async (req, res) => {
  await client.db().command({ ping: 1 });
  res.send("ok");
});
app.post("/api/users", async (req, res) => {
  const r = await users.insertOne({ name: req.body.name, city: req.body.city });
  res.status(201).json({ id: r.insertedId, ...req.body });
});
app.get("/api/users/:id", async (req, res) => {
  if (!ObjectId.isValid(req.params.id)) return res.status(400).send("bad id");
  const u = await users.findOne({ _id: new ObjectId(req.params.id) });
  u ? res.json(u) : res.status(404).send("no such user");
});

client.connect().then(() => {
  users = client.db().collection("users");
  app.listen(3000, () => console.log("users on 3000"));
});

users/Dockerfile

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY index.js .
USER node
EXPOSE 3000
CMD ["node", "index.js"]

orders/app.py

import os, httpx, psycopg
from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel

DB = os.environ["DATABASE_URL"]
USERS, PAY = os.environ["USERS_URL"], os.environ["PAYMENTS_URL"]
http = httpx.Client(timeout=httpx.Timeout(2.0, connect=1.0))
app = FastAPI()

@app.on_event("startup")
def init():
    with psycopg.connect(DB) as c:
        c.execute("CREATE TABLE IF NOT EXISTS orders (id SERIAL PRIMARY KEY, user_id TEXT, "
                  "item TEXT, amount INT, status TEXT, created_at TIMESTAMPTZ DEFAULT now())")

class NewOrder(BaseModel):
    user_id: str
    item: str
    amount: int

@app.get("/healthz")
def health():
    with psycopg.connect(DB) as c:
        c.execute("SELECT 1")
    return "ok"

@app.post("/api/orders", status_code=201)
def create(o: NewOrder, req: Request):
    rid = {"x-request-id": req.headers.get("x-request-id", "none")}
    print({"svc": "orders", "rid": rid["x-request-id"], "step": "start"}, flush=True)
    try:
        u = http.get(f"{USERS}/api/users/{o.user_id}", headers=rid)
    except httpx.HTTPError:
        raise HTTPException(503, "users service unavailable")
    if u.status_code != 200:
        raise HTTPException(400, "unknown user")
    try:
        p = http.post(f"{PAY}/charge", json={"amount": o.amount}, headers=rid)
    except httpx.HTTPError:
        raise HTTPException(503, "payments service unavailable, order not placed")
    status = "PAID" if p.status_code == 200 else "DECLINED"
    with psycopg.connect(DB) as c:
        oid = c.execute("INSERT INTO orders (user_id, item, amount, status) VALUES (%s,%s,%s,%s) RETURNING id",
                        (o.user_id, o.item, o.amount, status)).fetchone()[0]
    return {"id": oid, "status": status, "user": u.json()["name"], "payment": p.json()}

@app.get("/api/orders/{oid}")
def one(oid: int):
    with psycopg.connect(DB) as c:
        r = c.execute("SELECT id, user_id, item, amount, status FROM orders WHERE id=%s", (oid,)).fetchone()
    if not r:
        raise HTTPException(404, "not found")
    return dict(zip(["id", "user_id", "item", "amount", "status"], r))

orders/Dockerfile

FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi==0.115.2 uvicorn==0.32.0 httpx==0.27.2 "psycopg[binary]==3.2.3"
COPY app.py .
RUN useradd -r app
USER app
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

payments/main.go

package main

import (
    "encoding/json"
    "flag"
    "log"
    "net/http"
    "os"
    "strconv"
)

var version = "v1"

func main() {
    health := flag.Bool("health", false, "check health and exit")
    flag.Parse()
    if *health {
        if _, err := http.Get("http://localhost:8080/healthz"); err != nil {
            os.Exit(1)
        }
        os.Exit(0)
    }
    limit, _ := strconv.Atoi(os.Getenv("FAIL_ABOVE"))
    http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { w.Write([]byte("ok")) })
    http.HandleFunc("/charge", func(w http.ResponseWriter, r *http.Request) {
        var in struct{ Amount int }
        json.NewDecoder(r.Body).Decode(&in)
        log.Printf(`{"svc":"payments","rid":"%s","amount":%d}`, r.Header.Get("X-Request-Id"), in.Amount)
        w.Header().Set("Content-Type", "application/json")
        if in.Amount > limit {
            w.WriteHeader(402)
            json.NewEncoder(w).Encode(map[string]string{"result": "declined", "version": version})
            return
        }
        json.NewEncoder(w).Encode(map[string]string{"result": "charged", "version": version})
    })
    log.Fatal(http.ListenAndServe(":8080", nil))
}

payments/Dockerfile

FROM golang:1.23-alpine AS build
WORKDIR /src
COPY go.mod main.go ./
ARG VERSION=v1
RUN CGO_ENABLED=0 go build -ldflags "-s -w -X main.version=${VERSION}" -o /payments .

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /payments /payments
EXPOSE 8080
ENTRYPOINT ["/payments"]

What each line does

• Each service has its own database — users owns Mongo, orders owns Postgres. Orders asks users over HTTP for a name; it never reads the Mongo collection.
• httpx Timeout(2.0, connect=1.0) — Orders waits at most 2 seconds for another service. A slow payments service cannot hang every order.
• 503 on HTTPError — A clear error tells the gateway and user that a dependency is down, instead of a crash.
• Pass x-request-id on — Every log line for one order has the same id in all three services.
• 402 for declined — Payments uses a real HTTP code; orders saves DECLINED instead of failing.
• -X main.version — The Go build stamps a version into the binary, so we can see which version answered.
• distroless nonroot — The payments image has no shell and runs as a non-root user. It is about 8 MB.

Green output

$ docker compose up -d --build
$ docker compose ps --format '{{.Service}} {{.Status}}'
gateway Up 10 seconds
orders Up 25 seconds (healthy)
orders-db Up 31 seconds (healthy)
payments Up 25 seconds (healthy)
users Up 25 seconds (healthy)
users-db Up 31 seconds (healthy)
$ docker images --format '{{.Repository}} {{.Size}}' | grep shop
shop-payments 8.6MB
shop-users 142MB
shop-orders 189MB

Expected result. Three services in three languages are healthy, each with its own image and its own data.

Example 3: The gateway: nginx routes paths to services, adds a request id, and we test the full order flow

Goal. The gateway: nginx routes paths to services, adds a request id, and we test the full order flow.

gateway/nginx.conf

upstream users    { server users:3000; }
upstream orders   { server orders:8000; }
upstream payments { server payments:8080; }

map $http_x_request_id $rid {
    default $http_x_request_id;
    ""      $request_id;
}

log_format gw '$remote_addr "$request" $status rid=$rid up=$upstream_addr t=$request_time';

server {
    listen 80;
    access_log /dev/stdout gw;
    proxy_set_header X-Request-Id $rid;
    proxy_connect_timeout 2s;
    proxy_read_timeout 5s;
    add_header X-Request-Id $rid always;

    location /api/users/    { proxy_pass http://users; }
    location /api/orders/   { proxy_pass http://orders; }
    location = /api/orders  { proxy_pass http://orders; }
    location = /api/users   { proxy_pass http://users; }
    location /api/payments/ { return 403 "payments is internal only\n"; }
    location = /healthz     { return 200 "gateway ok\n"; }
    location /              { return 404 "no such api\n"; }
}

ops/order-flow.sh

U=$(curl -s -X POST localhost:8080/api/users -H 'content-type: application/json' \
    -d '{"name":"Ravi","city":"Pune"}' | python3 -c "import json,sys; print(json.load(sys.stdin)['id'])")
echo "user id: $U"
curl -s -i -X POST localhost:8080/api/orders -H 'content-type: application/json' \
    -d "{\"user_id\":\"$U\",\"item\":\"Redmi Note 13\",\"amount\":17999}"
echo
curl -s -X POST localhost:8080/api/orders -H 'content-type: application/json' \
    -d "{\"user_id\":\"$U\",\"item\":\"MacBook Air\",\"amount\":99900}"
echo

What each line does

• upstream users — Docker DNS turns users into the container IP. With --scale users=3 nginx gets all three IPs.
• map ... $rid — Keep the client request id if it sent one; otherwise nginx makes a new random one.
• proxy_set_header X-Request-Id — Every service receives the same id.
• proxy_read_timeout 5s — The gateway gives up after 5 seconds and returns 504, instead of waiting forever.
• payments returns 403 — Customers must never call payments directly. Only orders calls it on the backend network.
• location / returns 404 — Unknown paths do not leak to any service.

Green output

$ bash ops/order-flow.sh
user id: 671b2f0c9e1a4d0012ab34cd
HTTP/1.1 201 Created
X-Request-Id: 5f0c3d9a8b7e6f1a2b3c4d5e6f708192
{"id":1,"status":"PAID","user":"Ravi","payment":{"result":"charged","version":"v1"}}
{"id":2,"status":"DECLINED","user":"Ravi","payment":{"result":"declined","version":"v1"}}
$ curl -s localhost:8080/api/payments/charge
payments is internal only
$ docker compose logs --no-log-prefix | grep 5f0c3d9a
{"svc":"users","rid":"5f0c3d9a8b7e6f1a2b3c4d5e6f708192","path":"/api/users/671b2f0c9e1a4d0012ab34cd"}
{'svc': 'orders', 'rid': '5f0c3d9a8b7e6f1a2b3c4d5e6f708192', 'step': 'start'}
{"svc":"payments","rid":"5f0c3d9a8b7e6f1a2b3c4d5e6f708192","amount":17999}
172.18.0.1 "POST /api/orders HTTP/1.1" 201 rid=5f0c3d9a8b7e6f1a2b3c4d5e6f708192 up=172.19.0.4:8000 t=0.041

Expected result. One request id shows the full path of an order through the gateway, orders, users and payments.

Example 4: Failure and independent deploys: stop payments, scale users, release payments v2 alone, and clean up

Goal. Failure and independent deploys: stop payments, scale users, release payments v2 alone, and clean up.

ops/failure.sh

docker compose stop payments
curl -s -X POST localhost:8080/api/orders -H 'content-type: application/json' \
  -d "{\"user_id\":\"$U\",\"item\":\"Boat earbuds\",\"amount\":1499}"
echo
curl -s localhost:8080/api/users/$U | head -c 60
echo
docker compose start payments

ops/deploy-payments-v2.sh

docker compose build --build-arg VERSION=v2 payments
docker compose up -d --no-deps payments
docker compose ps payments
curl -s -X POST localhost:8080/api/orders -H 'content-type: application/json' \
  -d "{\"user_id\":\"$U\",\"item\":\"Mi Band\",\"amount\":2999}"
echo

ops/cleanup.sh

docker compose down -v --rmi local
docker network ls | grep shop || echo "networks gone"

What each line does

• docker compose stop payments — Only payments is down. Users and the gateway keep working.
• 503 payments unavailable — Orders fails fast and clearly. It does not save a half order.
• --build-arg VERSION=v2 — Builds a new payments image with a new version stamp.
• up -d --no-deps payments — Recreates only payments. Orders, users and both databases are not restarted.
• --scale users=3 — Run docker compose up -d --scale users=3 --no-recreate, then docker compose restart gateway so nginx reads the new IPs.
• down -v --rmi local — Removes containers, networks, both database volumes and the images we built.

Green output

$ bash ops/failure.sh
{"detail":"payments service unavailable, order not placed"}
{"_id":"671b2f0c9e1a4d0012ab34cd","name":"Ravi","city"
$ bash ops/deploy-payments-v2.sh
NAME             STATUS
shop-payments-1  Up 4 seconds (health: starting)
{"id":3,"status":"PAID","user":"Ravi","payment":{"result":"charged","version":"v2"}}
$ docker compose ps orders users --format '{{.Service}} {{.Status}}'
orders Up 14 minutes (healthy)
users Up 14 minutes (healthy)
$ bash ops/cleanup.sh
networks gone

Expected result. Payments broke without breaking users, and payments v2 went live while orders and users kept running for 14 minutes.

Every command, step by step (plain docker and Compose)

1. Start everything

   docker compose up -d --build
   

   What it does: Builds three images and starts six containers.

   Container shop-gateway-1 Started
   

2. Check health

   docker compose ps
   

   What it does: All services show healthy.

   orders Up (healthy)
   

3. Gateway check

   curl -s localhost:8080/healthz
   

   What it does: nginx answers.

   gateway ok
   

4. Create a user

   curl -s -X POST localhost:8080/api/users -H 'content-type: application/json' -d '{"name":"Ravi","city":"Pune"}'
   

   What it does: users writes to MongoDB.

   {"id":"671b...","name":"Ravi","city":"Pune"}
   

5. Place an order

   bash ops/order-flow.sh
   

   What it does: orders checks the user and calls payments.

   {"id":1,"status":"PAID"...}
   

6. Follow one request

   docker compose logs --no-log-prefix | grep RID
   

   What it does: The same id in all services.

   svc users ... svc orders ... svc payments
   

7. Block internal API

   curl -s localhost:8080/api/payments/charge
   

   What it does: Customers cannot call payments.

   payments is internal only
   

8. Prove network isolation

   docker compose exec gateway ping -c1 -W1 orders-db
   

   What it does: The gateway cannot even resolve the database.

   ping: bad address 'orders-db'
   

9. Deploy one service

   bash ops/deploy-payments-v2.sh
   

   What it does: Only payments restarts.

   "version":"v2"
   

10. Clean up

    bash ops/cleanup.sh
    

    What it does: Removes containers, volumes, networks and images.

    networks gone
    

When it is red 1

502 Bad Gateway  ... connect() failed (111: Connection refused) while connecting to upstream

Cause. The service behind the path is down or still starting.

Fix, step by step

1. docker compose ps to see which one is unhealthy.
2. docker compose logs SERVICE for its error.

When it is red 2

nginx: [emerg] host not found in upstream "payments:8080"

Cause. nginx started before the payments container existed, or the gateway is not on the backend network.

Fix, step by step

1. Keep depends_on with service_healthy.
2. Check the gateway networks include backend.

When it is red 3

{"detail":"users service unavailable"}

Cause. orders cannot reach users within 1 second.

Fix, step by step

1. docker compose exec orders python -c "import urllib.request as u; print(u.urlopen('http://users:3000/healthz').read())"
2. Check both are on the backend network.

When it is red 4

MongoServerSelectionError: getaddrinfo ENOTFOUND users-db

Cause. users is not on the users-data network.

Fix, step by step

1. Add users-data to the networks list of users.

When it is red 5

After --scale users=3 only one users container gets traffic

Cause. nginx resolved users once at start.

Fix, step by step

1. docker compose restart gateway, or use resolver 127.0.0.11 with a variable in proxy_pass.

Practice tasks

docker compose up -d --build
docker compose ps

bash ops/order-flow.sh

amount 99900

curl -H 'X-Request-Id: test-123' ...
docker compose logs | grep test-123

docker compose exec orders python -c "import socket; socket.gethostbyname('users-db')"

bash ops/failure.sh

docker compose up -d --scale users=3 --no-recreate
docker compose restart gateway

bash ops/deploy-payments-v2.sh

location /api/catalog/ { proxy_pass http://catalog; }

bash ops/cleanup.sh

Recap

• A microservice owns one job and its own data, and deploys alone.
• One nginx gateway is the only public door; it routes by path.
• Services find each other by Compose service name on a shared network.
• Put each database on an internal network that only its owner joins.
• Use timeouts and clear 503 errors between services; pass a request id for tracing.
• Update one service with up -d --no-deps; clean up with down -v.

FAQ

Q. Are microservices always better?

No. For a small team, one well-built app is simpler. Split when teams and deploys start blocking each other.

Q. Can services share one database?

They can, but then they are tied together again. Database per service is the clean rule.

Q. What about Kong, Traefik or AWS API Gateway?

They are gateways with more features such as auth plugins and rate limits. nginx is enough to learn the idea. Traefik comes in Project 23.

Q. How do I trace better than grep?

Use OpenTelemetry with Jaeger or Grafana Tempo. The request id is the first step.

Q. What if payments must not be called twice?

Send an idempotency key with each charge, so a retry does not charge again.

Q. How do services talk without waiting?

Use a queue or events (Projects 24 and 25) instead of direct HTTP.

Interview questions

1. How do services discover each other in Docker Compose?

   Model answer: Through Docker's embedded DNS: each service name resolves to its container IPs on shared networks.

2. What is an API gateway and why use one?

   Model answer: A single entry point that routes, hides internal services, and adds auth, limits and logging in one place.

3. How do you stop one failing service from bringing down others?

   Model answer: Timeouts, clear error responses, retries with limits, circuit breakers, and health checks.

4. How do you deploy one microservice without touching others?

   Model answer: Build and push only its image, then docker compose up -d --no-deps or a rolling update for that service.

5. Why database per service?

   Model answer: Loose coupling: each team owns its schema and data, and others use its API.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview card

मराठी cards