Chapter 58: Docker Projects Part 2: AWS RDS, DynamoDB, S3, Reel Upload, Microservices, Queues, Logging, Monitoring and Cloud Deploys

58.3 Project 19: Containers with S3 and RDS (image-upload API with presigned URLs, MinIO locally, real S3 and RDS on AWS)

By Ravindra Bagale · Last updated: 25 October 2026

Why: the problem

Suppose we work at Swiggy. Restaurants upload photos of their dishes. Every day thousands of photos come in. Where do we keep them? Not inside the container. A container can be deleted at any time, and its files go with it. Not on the EC2 disk either. If we run three copies of the API, each copy has a different disk. So we keep the photo files in Amazon S3, which is object storage that never fills up. We keep the facts about each photo (owner, size, S3 key, time) in a database. Here that is Amazon RDS MySQL. There is one more trick. The photo should not pass through our API at all. The API gives the browser a presigned URL. The browser uploads straight to S3 with that URL. Our container stays small and fast. On the laptop we use MinIO, an S3-compatible server in a container, and a MySQL container. On AWS we change only the endpoint and the database host. At the end we empty and delete the bucket, delete the RDS database and the IAM role, so nothing keeps billing.

What: every term in plain English

• Amazon S3 — Object storage. You put files (objects) in buckets and read them by key, like photos/42/misal.jpg.
• Object key — The full name of a file inside a bucket. There are no real folders; slashes are only part of the name.
• Presigned URL — A link signed by our API that lets anyone holding it do one action (PUT or GET) on one key for a short time.
• MinIO — An S3-compatible server that runs in one container. The AWS SDKs talk to it like real S3.
• Amazon RDS MySQL — A managed MySQL server. AWS does backups, patches and failover.
• Metadata table — A table that stores the facts about each file: id, owner, key, size, content type, status.
• Block Public Access — An S3 setting that stops anyone from making the bucket public by mistake. Keep it on.
• Instance role — An IAM role on EC2. Containers get short-lived credentials from it, so no keys are written anywhere.

How: step by step

1. Start MinIO and MySQL in Compose. MinIO on 9000 (API) and 9001 (console).
2. Create the bucket with a one-shot mc container. mc mb local/photos.
3. Start the API. FastAPI with boto3 and PyMySQL; env vars choose MinIO or AWS.
4. Ask for an upload URL, then PUT the file. The file goes straight to storage, not through the API.
5. Confirm the upload and list photos. The API checks the object with head_object and saves the size in MySQL.
6. Move to AWS. Real bucket, RDS MySQL, an instance role, and the same image.
7. Clean up. Empty and delete the bucket, delete RDS and the role.

Architecture

Folder layout

photo-store/
  compose.yaml          (minio + make-bucket + db)
  compose.local.yaml    (api with S3_ENDPOINT)
  compose.aws.yaml      (migrate + api, no keys)
  db/init.sql
  api/app.py
  api/Dockerfile
  aws/01-bucket.sh  aws/cors.json  aws/lifecycle.json
  aws/02-rds.sh
  aws/03-policy.json  aws/04-role.sh
  aws/get-secret.sh  aws/create-app-user.sql
  aws/99-cleanup.sh

Example 1: Local storage: MinIO and MySQL in Compose, plus a one-shot container that creates the bucket

Goal. Local storage: MinIO and MySQL in Compose, plus a one-shot container that creates the bucket.

compose.yaml

services:
  minio:
    image: minio/minio:RELEASE.2024-10-13T13-34-11Z
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: change-me-minio
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      retries: 10

  make-bucket:
    image: minio/mc:RELEASE.2024-10-08T09-37-26Z
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: ["sh", "-c", "mc alias set local http://minio:9000 minioadmin change-me-minio && mc mb --ignore-existing local/photos && echo bucket ready"]
    restart: "no"

  db:
    image: mysql:8.4
    environment:
      MYSQL_ROOT_PASSWORD: change-me-root
      MYSQL_DATABASE: photos
      MYSQL_USER: app
      MYSQL_PASSWORD: change-me-app
    volumes:
      - db-data:/var/lib/mysql
      - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 5s
      retries: 20

volumes:
  minio-data:
  db-data:

db/init.sql

CREATE TABLE IF NOT EXISTS photos (
  id           BIGINT AUTO_INCREMENT PRIMARY KEY,
  owner        VARCHAR(64)  NOT NULL,
  s3_key       VARCHAR(255) NOT NULL UNIQUE,
  content_type VARCHAR(64)  NOT NULL,
  size_bytes   BIGINT       NULL,
  status       VARCHAR(16)  NOT NULL DEFAULT 'PENDING',
  created_at   TIMESTAMP    DEFAULT CURRENT_TIMESTAMP,
  INDEX idx_owner (owner)
);

What each line does

• minio server /data — Stores objects in /data, which is a named volume, so files survive docker compose down.
• --console-address :9001 — The MinIO web console. Open localhost:9001 and log in with the placeholder user.
• make-bucket — A one-shot mc (MinIO client) container. It creates the photos bucket and exits with code 0.
• --ignore-existing — Running the job twice does not fail. Safe to re-run.
• init.sql in docker-entrypoint-initdb.d — MySQL runs it only on the first start, when the volume is empty.
• status PENDING — A row is created before the upload. It becomes UPLOADED only after we check the object exists.
• change-me-* — Placeholder passwords. Put real ones in a .env file that is never committed.

Green output

$ docker compose up -d minio db
$ docker compose run --rm make-bucket
Added `local` successfully.
Bucket created successfully `local/photos`.
bucket ready
$ docker compose exec db mysql -uapp -pchange-me-app photos -e "SHOW TABLES"
Tables_in_photos
photos

Expected result. Object storage and a database run on the laptop with zero AWS cost. The bucket and table are ready.

Example 2: The photo API in Python: presigned PUT URL, confirm step, presigned GET for viewing, and an env switch for MinIO or AWS

Goal. The photo API in Python: presigned PUT URL, confirm step, presigned GET for viewing, and an env switch for MinIO or AWS.

api/app.py

import os, uuid
import boto3, pymysql
from botocore.config import Config
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

BUCKET = os.environ["BUCKET"]
kw = {"region_name": os.environ["AWS_REGION"],
      "config": Config(signature_version="s3v4", s3={"addressing_style": "path"})}
if os.getenv("S3_ENDPOINT"):
    kw["endpoint_url"] = os.environ["S3_ENDPOINT"]
s3 = boto3.client("s3", **kw)
ALLOWED = {"image/jpeg": "jpg", "image/png": "png", "image/webp": "webp"}

def db():
    return pymysql.connect(host=os.environ["DB_HOST"], user=os.environ["DB_USER"],
                           password=os.environ["DB_PASSWORD"], database="photos",
                           autocommit=True, cursorclass=pymysql.cursors.DictCursor)

app = FastAPI()

class NewPhoto(BaseModel):
    owner: str
    content_type: str

@app.get("/healthz")
def health():
    with db() as c, c.cursor() as cur:
        cur.execute("SELECT 1")
    s3.head_bucket(Bucket=BUCKET)
    return "ok"

@app.post("/photos", status_code=201)
def new_photo(p: NewPhoto):
    if p.content_type not in ALLOWED:
        raise HTTPException(400, "only jpeg, png or webp")
    key = f"photos/{p.owner}/{uuid.uuid4().hex}.{ALLOWED[p.content_type]}"
    with db() as c, c.cursor() as cur:
        cur.execute("INSERT INTO photos (owner, s3_key, content_type) VALUES (%s,%s,%s)",
                    (p.owner, key, p.content_type))
        pid = cur.lastrowid
    url = s3.generate_presigned_url("put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": p.content_type},
        ExpiresIn=300)
    return {"id": pid, "key": key, "upload_url": url}

@app.post("/photos/{pid}/confirm")
def confirm(pid: int):
    with db() as c, c.cursor() as cur:
        cur.execute("SELECT s3_key FROM photos WHERE id=%s", (pid,))
        row = cur.fetchone()
        if not row:
            raise HTTPException(404, "no such photo")
        try:
            head = s3.head_object(Bucket=BUCKET, Key=row["s3_key"])
        except s3.exceptions.ClientError:
            raise HTTPException(409, "file not uploaded yet")
        cur.execute("UPDATE photos SET status='UPLOADED', size_bytes=%s WHERE id=%s",
                    (head["ContentLength"], pid))
    return {"id": pid, "status": "UPLOADED", "size_bytes": head["ContentLength"]}

@app.get("/photos/{pid}")
def view(pid: int):
    with db() as c, c.cursor() as cur:
        cur.execute("SELECT * FROM photos WHERE id=%s AND status='UPLOADED'", (pid,))
        row = cur.fetchone()
    if not row:
        raise HTTPException(404, "not found")
    row["view_url"] = s3.generate_presigned_url("get_object",
        Params={"Bucket": BUCKET, "Key": row["s3_key"]}, ExpiresIn=600)
    row["created_at"] = str(row["created_at"])
    return row

api/Dockerfile

FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi==0.115.2 uvicorn==0.32.0 boto3==1.35.40 pymysql==1.1.1 cryptography==43.0.1
COPY app.py .
RUN useradd -r app
USER app
EXPOSE 8000
HEALTHCHECK --interval=15s --timeout=3s CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

compose.local.yaml

services:
  api:
    build: ./api
    environment:
      BUCKET: photos
      AWS_REGION: ap-south-1
      S3_ENDPOINT: http://localhost:9000
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: change-me-minio
      DB_HOST: db
      DB_USER: app
      DB_PASSWORD: change-me-app
    network_mode: host
    depends_on:
      db:
        condition: service_healthy
      minio:
        condition: service_healthy

What each line does

• ExpiresIn=300 — The upload link works for only 5 minutes. A leaked link soon becomes useless.
• ContentType in Params — The signature includes the type, so the client must send the same Content-Type header.
• uuid4 in the key — Two users uploading misal.jpg never overwrite each other, and keys cannot be guessed.
• addressing_style path — MinIO works best with path style URLs: host/bucket/key.
• head_object in confirm — The API trusts S3, not the client. It saves the real size from S3.
• S3_ENDPOINT http://localhost:9000 — The presigned URL contains this host. The browser on the laptop must reach it, so we use localhost and host networking in this local test. On Docker Desktop, set S3_ENDPOINT to http://host.docker.internal:9000 and add it to your hosts file instead.
• HEALTHCHECK — Docker marks the container unhealthy if MySQL or the bucket cannot be reached.

Green output

$ docker compose -f compose.yaml -f compose.local.yaml up -d --build api
$ curl -s -X POST localhost:8000/photos -H 'content-type: application/json' -d '{"owner":"rest-42","content_type":"image/jpeg"}' | tee /tmp/p.json
{"id":1,"key":"photos/rest-42/9f1c...e2.jpg","upload_url":"http://localhost:9000/photos/photos/rest-42/9f1c...e2.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&..."}
$ URL=$(python3 -c "import json;print(json.load(open('/tmp/p.json'))['upload_url'])")
$ curl -s -X PUT -H 'Content-Type: image/jpeg' --data-binary @misal.jpg "$URL" -o /dev/null -w '%{http_code}\n'
200
$ curl -s -X POST localhost:8000/photos/1/confirm
{"id":1,"status":"UPLOADED","size_bytes":48213}
$ curl -s localhost:8000/photos/1 | python3 -m json.tool | grep -E 'status|view_url'
    "status": "UPLOADED",
    "view_url": "http://localhost:9000/photos/photos/rest-42/9f1c...e2.jpg?X-Amz-..."

Expected result. The photo bytes went straight to MinIO. The API only signed URLs and wrote one row in MySQL.

Example 3: Real AWS, part 1: a private bucket with CORS and a lifecycle rule, an RDS MySQL database, and a least-privilege instance role

Goal. Real AWS, part 1: a private bucket with CORS and a lifecycle rule, an RDS MySQL database, and a least-privilege instance role.

aws/01-bucket.sh

export AWS_REGION=ap-south-1
B=swiggy-photos-demo-111122223333
aws s3api create-bucket --bucket $B --create-bucket-configuration LocationConstraint=ap-south-1
aws s3api put-public-access-block --bucket $B \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
aws s3api put-bucket-encryption --bucket $B \
  --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
aws s3api put-bucket-cors --bucket $B --cors-configuration file://aws/cors.json
aws s3api put-bucket-lifecycle-configuration --bucket $B --lifecycle-configuration file://aws/lifecycle.json

aws/cors.json

{
  "CORSRules": [
    { "AllowedOrigins": ["https://partner.example.com"],
      "AllowedMethods": ["PUT", "GET"],
      "AllowedHeaders": ["Content-Type"],
      "MaxAgeSeconds": 3000 }
  ]
}

aws/lifecycle.json

{
  "Rules": [
    { "ID": "abort-half-uploads", "Status": "Enabled", "Filter": { "Prefix": "" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } },
    { "ID": "old-to-ia", "Status": "Enabled", "Filter": { "Prefix": "photos/" },
      "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" } ] }
  ]
}

aws/02-rds.sh

aws rds create-db-instance --db-instance-identifier photos-db \
  --engine mysql --engine-version 8.4.3 --db-instance-class db.t3.micro \
  --allocated-storage 20 --db-name photos \
  --master-username admin --manage-master-user-password \
  --vpc-security-group-ids sg-0db1234567890abcd \
  --no-publicly-accessible --backup-retention-period 1 \
  --tags Key=project,Value=docker-course
aws rds wait db-instance-available --db-instance-identifier photos-db
aws rds describe-db-instances --db-instance-identifier photos-db \
  --query 'DBInstances[0].Endpoint.Address' --output text

aws/03-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": "arn:aws:s3:::swiggy-photos-demo-111122223333/photos/*" },
    { "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::swiggy-photos-demo-111122223333" },
    { "Effect": "Allow",
      "Action": ["ssm:GetParameter"],
      "Resource": "arn:aws:ssm:ap-south-1:111122223333:parameter/photos/db-password" }
  ]
}

aws/04-role.sh

aws iam create-role --role-name photos-host-role \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam put-role-policy --role-name photos-host-role --policy-name photos-access \
  --policy-document file://aws/03-policy.json
aws iam create-instance-profile --instance-profile-name photos-host-profile
aws iam add-role-to-instance-profile --instance-profile-name photos-host-profile --role-name photos-host-role
aws ec2 associate-iam-instance-profile --instance-id i-0abc1234def567890 \
  --iam-instance-profile Name=photos-host-profile
aws ec2 modify-instance-metadata-options --instance-id i-0abc1234def567890 \
  --http-tokens required --http-put-response-hop-limit 2
aws ssm put-parameter --name /photos/db-password --type SecureString --value 'change-me-app'

What each line does

• Block Public Access, all four true — The bucket can never become public. Viewers use presigned GET URLs instead.
• SSE AES256 — Every object is encrypted at rest by S3 with no extra cost.
• CORS AllowedOrigins — A browser page on partner.example.com may PUT to the bucket. Without CORS the browser blocks the upload.
• AbortIncompleteMultipartUpload — Half-finished uploads are deleted after a day, so they do not cost money in secret.
• --manage-master-user-password — RDS keeps the admin password in Secrets Manager. We never type it.
• --no-publicly-accessible — Only the EC2 security group can reach port 3306. The database has no public IP.
• photos/* only — The role can put and get objects only under photos/. It cannot delete or touch other buckets.
• SSM SecureString — The app password is stored encrypted. The value here is a placeholder.

Green output

$ bash aws/01-bucket.sh
{ "Location": "http://swiggy-photos-demo-111122223333.s3.amazonaws.com/" }
$ bash aws/02-rds.sh
photos-db.abcdefghij12.ap-south-1.rds.amazonaws.com
$ bash aws/04-role.sh
$ aws s3api get-public-access-block --bucket swiggy-photos-demo-111122223333 --query 'PublicAccessBlockConfiguration.BlockPublicPolicy'
true

Expected result. A private, encrypted bucket and a private RDS database exist. The EC2 host has a role that can only touch photos/.

Example 4: Real AWS, part 2: run the same image on EC2 with no keys, create the table on RDS with a one-shot container, test, and clean up everything

Goal. Real AWS, part 2: run the same image on EC2 with no keys, create the table on RDS with a one-shot container, test, and clean up everything.

compose.aws.yaml

services:
  migrate:
    image: mysql:8.4
    entrypoint: ["sh", "-c", "mysql -h $$DB_HOST -u app -p$$DB_PASSWORD photos -e 'source /init.sql'"]
    environment:
      DB_HOST: photos-db.abcdefghij12.ap-south-1.rds.amazonaws.com
      DB_PASSWORD: ${DB_PASSWORD:?run get-secret.sh first}
    volumes:
      - ./db/init.sql:/init.sql:ro
    restart: "no"

  api:
    build: ./api
    environment:
      BUCKET: swiggy-photos-demo-111122223333
      AWS_REGION: ap-south-1
      DB_HOST: photos-db.abcdefghij12.ap-south-1.rds.amazonaws.com
      DB_USER: app
      DB_PASSWORD: ${DB_PASSWORD:?run get-secret.sh first}
    ports:
      - "8000:8000"
    restart: unless-stopped

aws/get-secret.sh

export DB_PASSWORD=$(aws ssm get-parameter --name /photos/db-password \
  --with-decryption --query Parameter.Value --output text)
echo "DB_PASSWORD loaded into this shell only"

aws/create-app-user.sql

CREATE USER IF NOT EXISTS 'app'@'%' IDENTIFIED BY 'change-me-app';
GRANT SELECT, INSERT, UPDATE ON photos.* TO 'app'@'%';

aws/99-cleanup.sh

B=swiggy-photos-demo-111122223333
docker compose -f compose.aws.yaml down --rmi local
docker compose -f compose.yaml -f compose.local.yaml down -v
aws s3 rm s3://$B --recursive
aws s3api delete-bucket --bucket $B
aws rds delete-db-instance --db-instance-identifier photos-db --skip-final-snapshot --delete-automated-backups
aws rds wait db-instance-deleted --db-instance-identifier photos-db
aws ssm delete-parameter --name /photos/db-password
aws ec2 disassociate-iam-instance-profile --association-id iip-assoc-0123456789abcdef0
aws iam remove-role-from-instance-profile --instance-profile-name photos-host-profile --role-name photos-host-role
aws iam delete-instance-profile --instance-profile-name photos-host-profile
aws iam delete-role-policy --role-name photos-host-role --policy-name photos-access
aws iam delete-role --role-name photos-host-role

What each line does

• No S3_ENDPOINT, no AWS keys — boto3 finds real S3 and takes credentials from the instance role through IMDSv2.
• ${DB_PASSWORD:?...} — Compose stops with a clear message if the password was not loaded. It is never written into a file.
• $$DB_HOST — Double dollar tells Compose to leave the variable for the container shell.
• migrate service — A one-shot container runs init.sql on RDS from inside the VPC. No laptop needs database access.
• GRANT SELECT, INSERT, UPDATE — The app user cannot DROP or DELETE. Create it once with the admin secret from Secrets Manager.
• aws s3 rm --recursive — A bucket must be empty before delete-bucket works.
• --skip-final-snapshot — Fine for practice. For real data, keep a final snapshot.

Green output

$ source aws/get-secret.sh
DB_PASSWORD loaded into this shell only
$ docker compose -f compose.aws.yaml run --rm migrate
$ docker compose -f compose.aws.yaml up -d --build api
$ curl -s -X POST localhost:8000/photos -H 'content-type: application/json' -d '{"owner":"rest-42","content_type":"image/png"}'
{"id":1,"key":"photos/rest-42/4b7d...a1.png","upload_url":"https://swiggy-photos-demo-111122223333.s3.ap-south-1.amazonaws.com/photos/rest-42/4b7d...a1.png?X-Amz-..."}
$ curl -s -X PUT -H 'Content-Type: image/png' --data-binary @thali.png "$URL" -w '%{http_code}\n'
200
$ curl -s -X POST localhost:8000/photos/1/confirm
{"id":1,"status":"UPLOADED","size_bytes":91822}
$ bash aws/99-cleanup.sh
$ aws s3 ls | grep swiggy-photos || echo "bucket gone"
bucket gone

Expected result. The same image ran on EC2 with real S3 and RDS, used no keys, and every AWS resource was deleted at the end.

Every command, step by step (plain docker and Compose)

1. Start storage

   docker compose up -d minio db
   

   What it does: Starts MinIO and MySQL with named volumes.

   Container photo-store-db-1 Healthy
   

2. Create the bucket

   docker compose run --rm make-bucket
   

   What it does: Creates the photos bucket and exits.

   bucket ready
   

3. Start the API

   docker compose -f compose.yaml -f compose.local.yaml up -d --build api
   

   What it does: Builds the Python image and starts it.

   Container photo-store-api-1 Started
   

4. Health check

   curl -s localhost:8000/healthz
   

   What it does: The API reaches MySQL and the bucket.

   "ok"
   

5. Ask for an upload URL

   curl -s -X POST localhost:8000/photos -H 'content-type: application/json' -d '{"owner":"rest-42","content_type":"image/jpeg"}'
   

   What it does: Creates a PENDING row and returns a 5-minute PUT URL.

   {"id":1,"key":"photos/rest-42/...jpg","upload_url":"http://localhost:9000/..."}
   

6. Upload the file

   curl -s -X PUT -H 'Content-Type: image/jpeg' --data-binary @misal.jpg "$URL" -w '%{http_code}'
   

   What it does: The file goes straight to storage.

   200
   

7. Confirm

   curl -s -X POST localhost:8000/photos/1/confirm
   

   What it does: The API checks the object and saves its size.

   {"id":1,"status":"UPLOADED","size_bytes":48213}
   

8. See the files

   docker compose run --rm --entrypoint sh make-bucket -c 'mc alias set local http://minio:9000 minioadmin change-me-minio && mc ls --recursive local/photos'
   

   What it does: Lists objects in MinIO.

   48KiB photos/rest-42/9f1c...e2.jpg
   

9. Move to AWS

   bash aws/01-bucket.sh && bash aws/02-rds.sh && bash aws/04-role.sh
   

   What it does: Private bucket, private RDS, role on EC2.

   photos-db.abcdefghij12.ap-south-1.rds.amazonaws.com
   

10. Clean up

    bash aws/99-cleanup.sh
    

    What it does: Empties and deletes the bucket, deletes RDS, the parameter and the role.

    bucket gone
    

When it is red 1

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.

Cause. The PUT sent a different Content-Type than the one signed, or the URL host was changed.

Fix, step by step

1. Send exactly the same Content-Type header that was given to the API.
2. Do not edit the URL. Copy it whole, with quotes in the shell.

When it is red 2

AccessDenied: Request has expired

Cause. The presigned URL is older than ExpiresIn (300 seconds).

Fix, step by step

1. Ask the API for a new URL.
2. Raise ExpiresIn only a little; short links are safer.

When it is red 3

Access to fetch at 'https://swiggy-photos-demo...' from origin 'https://partner.example.com' has been blocked by CORS policy

Cause. The bucket has no CORS rule for that website.

Fix, step by step

1. Add the origin to aws/cors.json and run put-bucket-cors again.

When it is red 4

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:9000/photos"

Cause. Inside a normal bridge network, localhost is the container itself, not MinIO.

Fix, step by step

1. Use host networking as in this lesson, or set S3_ENDPOINT to an address that both the container and the browser can reach.

When it is red 5

pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'photos-db....rds.amazonaws.com' (timed out)")

Cause. The RDS security group does not allow port 3306 from the EC2 security group.

Fix, step by step

1. Add an inbound rule on sg-0db1234567890abcd: MySQL 3306 from the EC2 security group id.

When it is red 6

409 file not uploaded yet

Cause. Confirm was called before the PUT finished, or the PUT failed.

Fix, step by step

1. Check the PUT returned 200, then call confirm again.

Practice tasks

docker compose up -d minio db
docker compose run --rm make-bucket

Log in with minioadmin and the placeholder password

curl -s -X PUT -H 'Content-Type: image/jpeg' --data-binary @misal.jpg "$URL"

SignatureDoesNotMatch

AccessDenied: Request has expired

curl -s localhost:8000/photos/1

400 only jpeg, png or webp

s3.delete_object(Bucket=BUCKET, Key=key)

See Project 20, the reel-upload app

bash aws/99-cleanup.sh
aws rds describe-db-instances --query 'DBInstances[].DBInstanceIdentifier'

Recap

• Never keep uploaded files inside a container; use S3 (or MinIO locally).
• Keep metadata in a database such as RDS MySQL.
• Presigned URLs let the browser upload straight to storage for a few minutes.
• Confirm uploads with head_object before trusting them.
• Keep the bucket private and encrypted; containers use an instance role, not keys.
• Empty and delete the bucket, and delete RDS and the role, after practice.

FAQ

Q. Is MinIO the same as S3?

It speaks the same API, so boto3 and the AWS CLI work with it. Some features like lifecycle classes differ.

Q. Can I upload big videos with one presigned PUT?

A single PUT goes up to 5 GB. For big files use multipart upload with presigned part URLs.

Q. Why not make the bucket public for viewing?

Public buckets leak data. Use presigned GET URLs or CloudFront with origin access control.

Q. Does the RDS free tier cover this?

A db.t3.micro with 20 GB is free for 12 months on new accounts. Delete it after practice anyway.

Q. Where does the RDS admin password live?

In Secrets Manager because of --manage-master-user-password. Read it only to create the app user.

Q. Can this run on ECS?

Yes. Use an ECS task role for S3 and inject the DB password from SSM into the task definition.

Interview questions

1. How do you let users upload files without passing them through your servers?

   Model answer: Generate a presigned PUT URL in the API; the client uploads directly to S3.

2. How do you secure an S3 bucket used by containers?

   Model answer: Block Public Access, default encryption, a role limited to a key prefix, and short-lived presigned URLs.

3. Where do you store file metadata and why?

   Model answer: In a database like RDS, because we need queries, owners, status and joins that S3 cannot do.

4. How do you test S3 code without AWS?

   Model answer: Run MinIO or LocalStack in Compose and override the endpoint URL.

5. A presigned upload fails with SignatureDoesNotMatch. What do you check?

   Model answer: The Content-Type header, the region, the clock on the server, and that the URL was not changed.

Samajla ka?

Ravi Sir's Cards for this lesson

Revision cardInterview card

मराठी cards