Chapter 58: Docker Projects Part 2: AWS RDS, DynamoDB, S3, Reel Upload, Microservices, Queues, Logging, Monitoring and Cloud Deploys

58.4 Project 20: Reel-upload app (upload to S3, Redis queue, ffmpeg worker container, thumbnails and a feed)

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we work at Instagram on the Reels team. A user uploads a 40 MB phone video. They want to see it in the feed soon. But the raw video is too big for phones on slow networks. We must make a small 720p copy and a thumbnail picture. That work (transcoding) takes 10 to 60 seconds of heavy CPU. If the upload API does it, the API freezes for every other user. So we split the job. The API only accepts the upload and puts a small message in a queue: "please process reel 7". A separate worker container, with ffmpeg inside, takes messages from the queue one by one, makes the 720p file and the thumbnail, and saves them. When many reels come in, we run more worker copies with one command. The API never slows down. We build it with Compose: an API, Redis as the queue, an ffmpeg worker, MinIO as S3, Postgres for the feed rows. On AWS, MinIO becomes S3 with the same code.

What: every term in plain English

• Transcoding — Changing a video into another size or format, for example 1080p to 720p H.264.
• ffmpeg — The free command-line tool that almost every video site uses to convert video and grab frames.
• Job queue — A list of work items. Producers push jobs; workers pop them. Redis lists are a simple queue.
• BRPOP / BLMOVE — Redis commands that wait for a job and take it. BLMOVE also keeps a copy in a processing list, so a crash does not lose it.
• Worker — A container with no web port. It only takes jobs and does them.
• Thumbnail — A small JPEG picture from one frame of the video, shown in the feed before play.
• Scale out — docker compose up --scale worker=3 runs three workers sharing one queue.
• Status field — Each reel row moves UPLOADED to PROCESSING to READY or FAILED.

How: step by step

1. Start Redis, Postgres and MinIO. One compose file, named volumes.
2. Start the API. POST /reels takes a file, saves raw/ in storage, adds a row and a job.
3. Start the worker. It pops a job, downloads raw, runs ffmpeg twice, uploads the results.
4. Watch the feed. GET /feed shows READY reels with thumbnail and video URLs.
5. Scale workers. Upload 6 reels and run 3 workers.
6. Clean up. down -v removes containers, volumes and test files.

Architecture

Folder layout

reels/
  compose.yaml        (redis, db, minio, make-bucket, api, worker)
  compose.aws.yaml    (real S3, no keys)
  db/init.sql
  api/app.py  api/Dockerfile
  worker/worker.py  worker/Dockerfile
  ops/scale.sh  ops/requeue.sh  ops/cleanup.sh
  aws/policy.json

Example 1: The stack: Redis queue, Postgres feed table, MinIO storage, API and worker in one compose

Goal. The stack: Redis queue, Postgres feed table, MinIO storage, API and worker in one compose.yaml.

compose.yaml

x-app: &app
  environment:
    REDIS_URL: redis://redis:6379/0
    DATABASE_URL: postgresql://reels:change-me-db@db:5432/reels
    S3_ENDPOINT: http://minio:9000
    PUBLIC_S3: http://localhost:9000
    AWS_ACCESS_KEY_ID: minioadmin
    AWS_SECRET_ACCESS_KEY: change-me-minio
    AWS_REGION: ap-south-1
    BUCKET: reels
  depends_on:
    redis: { condition: service_healthy }
    db: { condition: service_healthy }
    minio: { condition: service_healthy }
  restart: unless-stopped

services:
  redis:
    image: redis:7.4-alpine
    command: ["redis-server", "--appendonly", "yes"]
    volumes: [redis-data:/data]
    healthcheck: { test: ["CMD", "redis-cli", "ping"], interval: 5s, retries: 10 }

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: reels
      POSTGRES_PASSWORD: change-me-db
      POSTGRES_DB: reels
    volumes:
      - db-data:/var/lib/postgresql/data
      - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    healthcheck: { test: ["CMD", "pg_isready", "-U", "reels"], interval: 5s, retries: 10 }

  minio:
    image: minio/minio:RELEASE.2024-10-13T13-34-11Z
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: change-me-minio
    volumes: [minio-data:/data]
    ports: ["9000:9000", "9001:9001"]
    healthcheck: { test: ["CMD", "mc", "ready", "local"], interval: 5s, retries: 10 }

  make-bucket:
    image: minio/mc:RELEASE.2024-10-08T09-37-26Z
    depends_on: { minio: { condition: service_healthy } }
    entrypoint: ["sh", "-c", "mc alias set m http://minio:9000 minioadmin change-me-minio && mc mb --ignore-existing m/reels && mc anonymous set download m/reels/out && echo bucket ready"]
    restart: "no"

  api:
    <<: *app
    build: ./api
    ports: ["8000:8000"]

  worker:
    <<: *app
    build: ./worker
    stop_grace_period: 90s
    deploy:
      resources:
        limits: { cpus: "1.0", memory: 768M }

volumes:
  redis-data:
  db-data:
  minio-data:

db/init.sql

CREATE TABLE IF NOT EXISTS reels (
  id         SERIAL PRIMARY KEY,
  owner      TEXT NOT NULL,
  caption    TEXT,
  raw_key    TEXT NOT NULL,
  video_key  TEXT,
  thumb_key  TEXT,
  status     TEXT NOT NULL DEFAULT 'UPLOADED',
  error      TEXT,
  created_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX IF NOT EXISTS reels_ready ON reels (status, created_at DESC);

What each line does

• x-app anchor — API and worker share the same settings, so they always point to the same queue, database and bucket.
• --appendonly yes — Redis writes every change to disk, so queued jobs survive a Redis restart.
• mc anonymous set download m/reels/out — Only the out/ prefix (finished files) can be read in this local test. raw/ stays private.
• PUBLIC_S3 — The address the browser uses. S3_ENDPOINT is the address containers use. They differ because localhost is not the same inside a container.
• stop_grace_period: 90s — When we stop a worker, Docker waits up to 90 seconds so the current video can finish.
• cpus 1.0, memory 768M — ffmpeg can eat every CPU. The limit protects Redis, Postgres and the API on the same host.
• reels_ready index — The feed query, READY reels newest first, stays fast.

Green output

$ docker compose up -d redis db minio
$ docker compose run --rm make-bucket
bucket ready
$ docker compose ps --format '{{.Service}} {{.Status}}'
db Up 20 seconds (healthy)
minio Up 20 seconds (healthy)
redis Up 20 seconds (healthy)

Expected result. Queue, database and storage are healthy. Next we add the code that uses them.

Example 2: The API (FastAPI) takes the upload, stores raw/ in S3, inserts a row and pushes a job

Goal. The API (FastAPI) takes the upload, stores raw/ in S3, inserts a row and pushes a job. The worker runs ffmpeg.

api/app.py

import os, uuid, json
import boto3, psycopg, redis
from fastapi import FastAPI, UploadFile, Form, HTTPException

s3 = boto3.client("s3", endpoint_url=os.getenv("S3_ENDPOINT"), region_name=os.environ["AWS_REGION"])
r = redis.Redis.from_url(os.environ["REDIS_URL"])
DB = os.environ["DATABASE_URL"]
BUCKET = os.environ["BUCKET"]
PUBLIC = os.getenv("PUBLIC_S3", "")
app = FastAPI()

@app.get("/healthz")
def health():
    r.ping()
    return "ok"

@app.post("/reels", status_code=202)
def upload(owner: str = Form(...), caption: str = Form(""), file: UploadFile = None):
    if file is None or not file.content_type.startswith("video/"):
        raise HTTPException(400, "send a video file")
    key = f"raw/{uuid.uuid4().hex}.mp4"
    s3.upload_fileobj(file.file, BUCKET, key, ExtraArgs={"ContentType": file.content_type})
    with psycopg.connect(DB) as c:
        rid = c.execute("INSERT INTO reels (owner, caption, raw_key) VALUES (%s,%s,%s) RETURNING id",
                        (owner, caption, key)).fetchone()[0]
    r.lpush("reels:jobs", json.dumps({"id": rid, "raw_key": key}))
    return {"id": rid, "status": "UPLOADED", "queued": r.llen("reels:jobs")}

@app.get("/reels/{rid}")
def one(rid: int):
    with psycopg.connect(DB) as c:
        row = c.execute("SELECT id, status, error FROM reels WHERE id=%s", (rid,)).fetchone()
    if not row:
        raise HTTPException(404, "not found")
    return {"id": row[0], "status": row[1], "error": row[2]}

@app.get("/feed")
def feed(limit: int = 20):
    with psycopg.connect(DB) as c:
        rows = c.execute("SELECT id, owner, caption, video_key, thumb_key FROM reels "
                         "WHERE status='READY' ORDER BY created_at DESC LIMIT %s", (limit,)).fetchall()
    return [{"id": i, "owner": o, "caption": cap,
             "video": f"{PUBLIC}/{BUCKET}/{v}", "thumb": f"{PUBLIC}/{BUCKET}/{t}"}
            for i, o, cap, v, t in rows]

api/Dockerfile

FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi==0.115.2 uvicorn==0.32.0 python-multipart==0.0.12 boto3==1.35.40 "psycopg[binary]==3.2.3" redis==5.1.1
COPY app.py .
RUN useradd -r app
USER app
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

worker/worker.py

import os, json, signal, subprocess, tempfile, socket
import boto3, psycopg, redis

s3 = boto3.client("s3", endpoint_url=os.getenv("S3_ENDPOINT"), region_name=os.environ["AWS_REGION"])
r = redis.Redis.from_url(os.environ["REDIS_URL"])
DB, BUCKET = os.environ["DATABASE_URL"], os.environ["BUCKET"]
ME = socket.gethostname()
running = True

def stop(*_):
    global running
    running = False
    print(ME, "got SIGTERM, finishing current job", flush=True)
signal.signal(signal.SIGTERM, stop)

def status(rid, s, **cols):
    sets = ", ".join(["status=%s"] + [f"{k}=%s" for k in cols])
    with psycopg.connect(DB) as c:
        c.execute(f"UPDATE reels SET {sets} WHERE id=%s", (s, *cols.values(), rid))

def process(job):
    rid, raw = job["id"], job["raw_key"]
    status(rid, "PROCESSING")
    with tempfile.TemporaryDirectory() as d:
        src, out, thumb = f"{d}/in.mp4", f"{d}/720.mp4", f"{d}/thumb.jpg"
        s3.download_file(BUCKET, raw, src)
        subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-i", src,
                        "-vf", "scale=-2:720", "-c:v", "libx264", "-preset", "veryfast",
                        "-crf", "28", "-c:a", "aac", "-b:a", "96k",
                        "-movflags", "+faststart", out], check=True, timeout=300)
        subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-ss", "1", "-i", src,
                        "-frames:v", "1", "-vf", "scale=360:-2", thumb], check=True, timeout=60)
        vkey, tkey = f"out/{rid}/720.mp4", f"out/{rid}/thumb.jpg"
        s3.upload_file(out, BUCKET, vkey, ExtraArgs={"ContentType": "video/mp4"})
        s3.upload_file(thumb, BUCKET, tkey, ExtraArgs={"ContentType": "image/jpeg"})
    status(rid, "READY", video_key=vkey, thumb_key=tkey)
    print(ME, "reel", rid, "READY", flush=True)

print(ME, "waiting for jobs", flush=True)
while running:
    raw = r.blmove("reels:jobs", "reels:processing", 5, "RIGHT", "LEFT")
    if raw is None:
        continue
    job = json.loads(raw)
    try:
        process(job)
    except Exception as e:
        status(job["id"], "FAILED", error=str(e)[:500])
        print(ME, "reel", job["id"], "FAILED", e, flush=True)
    finally:
        r.lrem("reels:processing", 1, raw)

worker/Dockerfile

FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /app
RUN pip install --no-cache-dir boto3==1.35.40 "psycopg[binary]==3.2.3" redis==5.1.1
COPY worker.py .
RUN useradd -r worker
USER worker
CMD ["python", "-u", "worker.py"]

What each line does

• status_code=202 — 202 Accepted means: we got it, the work happens later. The user does not wait for ffmpeg.
• lpush reels:jobs — The API puts a small JSON job on the left of the list. The worker takes from the right, so it is first in, first out.
• blmove ... reels:processing — The job moves to a second list while it runs. If the worker dies, the job is still in reels:processing and can be re-queued.
• scale=-2:720 — Height 720, width auto and even. H.264 needs even sizes.
• -crf 28 -preset veryfast — Smaller file and faster encode. Fine for phone viewing.
• +faststart — Moves video info to the front, so the phone starts playing before the full download.
• -ss 1 -frames:v 1 — Takes one frame at second 1 as the thumbnail.
• SIGTERM handler — docker compose stop sends SIGTERM. The loop ends after the current job, so no reel is half done.
• timeout=300 — A broken video cannot block the worker forever.

Green output

$ docker compose up -d --build api worker
$ docker run --rm -v "$PWD":/w -w /w jrottenberg/ffmpeg:7.0-alpine -f lavfi -i testsrc=duration=5:size=1920x1080:rate=30 -f lavfi -i sine=duration=5 -shortest test.mp4
$ curl -s -F owner=priya -F caption="Pune rain" -F file=@test.mp4\;type=video/mp4 localhost:8000/reels
{"id":1,"status":"UPLOADED","queued":1}
$ docker compose logs -f worker
worker-1  | 3f2a9c1d7e44 waiting for jobs
worker-1  | 3f2a9c1d7e44 reel 1 READY
$ curl -s localhost:8000/feed
[{"id":1,"owner":"priya","caption":"Pune rain","video":"http://localhost:9000/reels/out/1/720.mp4","thumb":"http://localhost:9000/reels/out/1/thumb.jpg"}]

Expected result. The upload returned at once. A separate ffmpeg container made the 720p video and thumbnail, and the feed shows the reel.

Example 3: Scale workers, survive a crash, look inside the queue, and move to real S3 on AWS

Goal. Scale workers, survive a crash, look inside the queue, and move to real S3 on AWS.

ops/scale.sh

for i in 1 2 3 4 5 6; do
  curl -s -F owner=demo -F caption="reel $i" -F file=@test.mp4\;type=video/mp4 localhost:8000/reels
  echo
done
docker compose up -d --scale worker=3 --no-recreate
docker compose exec redis redis-cli LLEN reels:jobs
docker compose logs --since 2m worker | grep READY

ops/requeue.sh

# move jobs left behind by a crashed worker back to the main queue
while [ "$(docker compose exec -T redis redis-cli LLEN reels:processing)" != "0" ]; do
  docker compose exec -T redis redis-cli LMOVE reels:processing reels:jobs RIGHT LEFT
done

compose.aws.yaml

services:
  api:
    environment:
      S3_ENDPOINT: ""
      PUBLIC_S3: https://d1234abcd.cloudfront.net
      AWS_ACCESS_KEY_ID: ""
      AWS_SECRET_ACCESS_KEY: ""
      BUCKET: insta-reels-demo-111122223333
  worker:
    environment:
      S3_ENDPOINT: ""
      AWS_ACCESS_KEY_ID: ""
      AWS_SECRET_ACCESS_KEY: ""
      BUCKET: insta-reels-demo-111122223333

aws/policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": ["arn:aws:s3:::insta-reels-demo-111122223333/raw/*",
                   "arn:aws:s3:::insta-reels-demo-111122223333/out/*"] }
  ]
}

ops/cleanup.sh

docker compose -f compose.yaml -f compose.aws.yaml down --rmi local
docker compose down -v
rm -f test.mp4
aws s3 rm s3://insta-reels-demo-111122223333 --recursive
aws s3api delete-bucket --bucket insta-reels-demo-111122223333

What each line does

• --scale worker=3 — Three worker containers share one Redis list. Each job goes to exactly one worker, because BLMOVE is atomic.
• --no-recreate — Running containers are not restarted, so jobs already in progress are safe.
• LLEN reels:jobs — How many reels are still waiting. If it keeps growing, add workers.
• requeue.sh — After docker kill of a worker, its job is still in reels:processing. This moves it back.
• Empty S3_ENDPOINT and keys — boto3 then uses real S3 and the EC2 instance role. The same images run unchanged.
• CloudFront in PUBLIC_S3 — Viewers get videos from a CDN close to them. The bucket itself stays private with origin access control.
• raw/* and out/* only — Workers can read and write only these prefixes.

Green output

$ bash ops/scale.sh
{"id":2,"status":"UPLOADED","queued":1}
...
{"id":7,"status":"UPLOADED","queued":6}
4
worker-2  | 8b1e0f2c3a91 reel 2 READY
worker-3  | c7d4a2e9b0f5 reel 3 READY
worker-1  | 3f2a9c1d7e44 reel 4 READY
$ docker kill reels-worker-2
$ docker compose exec redis redis-cli LRANGE reels:processing 0 -1
1) "{\"id\": 6, \"raw_key\": \"raw/a1b2....mp4\"}"
$ bash ops/requeue.sh && docker compose up -d --scale worker=3
$ curl -s localhost:8000/reels/6
{"id":6,"status":"READY","error":null}
$ bash ops/cleanup.sh

Expected result. Three workers shared the load, a killed worker lost no job, and the same images are ready for real S3 with a role.

Every command, step by step (plain docker and Compose)

1. Start the backing services

   docker compose up -d redis db minio
   

   What it does: Queue, database and storage start with volumes.

   redis Up (healthy)
   

2. Create the bucket

   docker compose run --rm make-bucket
   

   What it does: Makes the reels bucket; out/ is readable locally.

   bucket ready
   

3. Start API and worker

   docker compose up -d --build api worker
   

   What it does: Builds two images: one with FastAPI, one with ffmpeg.

   Container reels-worker-1 Started
   

4. Make a test video

   docker run --rm -v "$PWD":/w -w /w jrottenberg/ffmpeg:7.0-alpine -f lavfi -i testsrc=duration=5:size=1920x1080:rate=30 -f lavfi -i sine=duration=5 -shortest test.mp4
   

   What it does: Creates a 5 second 1080p video with sound.

   test.mp4 created
   

5. Upload

   curl -s -F owner=priya -F file=@test.mp4\;type=video/mp4 localhost:8000/reels
   

   What it does: Returns at once with 202.

   {"id":1,"status":"UPLOADED","queued":1}
   

6. Watch the worker

   docker compose logs -f worker
   

   What it does: The worker takes the job and runs ffmpeg.

   reel 1 READY
   

7. Check the feed

   curl -s localhost:8000/feed
   

   What it does: READY reels with video and thumbnail URLs.

   [{"id":1,..."thumb":"http://localhost:9000/reels/out/1/thumb.jpg"}]
   

8. Check the file size

   docker compose exec worker python -c "print('ok')" && curl -sI localhost:9000/reels/out/1/720.mp4 | grep -i content-length
   

   What it does: The 720p copy is much smaller than the raw file.

   Content-Length: 412870
   

9. Scale

   docker compose up -d --scale worker=3 --no-recreate
   

   What it does: Three workers share the queue.

   Container reels-worker-3 Started
   

10. Clean up

    bash ops/cleanup.sh
    

    What it does: Removes containers, images, volumes and the test video.

    Volume reels_minio-data Removed
    

When it is red 1

FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg'

Cause. ffmpeg was not installed in the worker image, or the API image was used for the worker.

Fix, step by step

1. Check worker/Dockerfile installs ffmpeg.
2. Check the worker service has build: ./worker.

When it is red 2

subprocess.CalledProcessError: Command '['ffmpeg', ...]' returned non-zero exit status 1.

Cause. The file is not a real video, or it is broken.

Fix, step by step

1. Run ffmpeg -i on the file by hand inside the worker with docker compose exec.
2. The reel is marked FAILED with the error, so the user can upload again.

When it is red 3

worker-1 exited with code 137

Cause. The worker hit the 768M memory limit (OOM kill) on a big video.

Fix, step by step

1. Raise memory a little, or add -threads 2 to ffmpeg.
2. Check with docker stats while a job runs.

When it is red 4

Jobs stay in reels:jobs and LLEN keeps growing

Cause. Workers are too slow or none are running.

Fix, step by step

1. docker compose ps worker to check they are up.
2. Scale with --scale worker=N, or move workers to a second host.

When it is red 5

The feed shows broken video links in the browser

Cause. PUBLIC_S3 points to http://minio:9000, which only works inside Docker.

Fix, step by step

1. Set PUBLIC_S3 to http://localhost:9000 locally, or the CloudFront domain on AWS.

Practice tasks

docker compose up -d redis db minio
docker compose ps

curl -s localhost:8000/reels/1

curl -s localhost:8000/feed

docker compose stop worker
docker compose exec redis redis-cli LLEN reels:jobs

docker compose up -d --scale worker=3

curl -s localhost:8000/reels/ID

docker stats --no-stream

docker kill reels-worker-1
bash ops/requeue.sh

-vf scale=-2:480

bash ops/cleanup.sh

Recap

• Heavy work like transcoding belongs in a worker, not in the API.
• The API returns 202 and puts a small job in a queue such as a Redis list.
• ffmpeg in its own image makes the 720p copy and the thumbnail.
• BLMOVE keeps jobs safe if a worker crashes; scale workers with --scale.
• Limit worker CPU and memory so it cannot starve other containers.
• The same images use real S3 by removing the endpoint and keys.

FAQ

Q. Is Redis a real queue for production?

It is fine for small and medium apps. Bigger systems use Amazon SQS, RabbitMQ or Kafka (Projects 24 and 25).

Q. Can workers use a GPU?

Yes. Use an ffmpeg build with NVENC and run the worker with gpus: all, as in Project 12.

Q. Why upload through the API here and presigned URLs in Project 19?

Here we keep it simple. In production, combine both: presigned upload, then the API queues the job on confirm.

Q. How do I get progress percent?

Parse ffmpeg -progress output in the worker and store it in Redis for the API to read.

Q. Can I use AWS MediaConvert instead?

Yes, it is a managed transcoder. The worker pattern is still useful for thumbnails and custom steps.

Q. Why stop_grace_period?

So docker compose stop or a deploy waits for the current video instead of killing it.

Interview questions

1. How would you design a video upload pipeline?

   Model answer: Upload to object storage, record metadata, push a job to a queue, transcode in scalable workers, serve through a CDN.

2. How do you make sure a queued job is not lost if a worker crashes?

   Model answer: Use an acknowledged pattern: BLMOVE to a processing list, SQS visibility timeout, or RabbitMQ acks.

3. How do you scale workers in Docker?

   Model answer: docker compose up --scale worker=N, or replicas in Swarm or Kubernetes, based on queue length.

4. Why put resource limits on CPU-heavy containers?

   Model answer: To protect other services on the same host and make performance predictable.

5. How do you stop a worker safely during a deploy?

   Model answer: Handle SIGTERM, finish the current job, and set a long enough stop grace period.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview card

मराठी cards