Ravindra BagaleCourses & study guides मराठी Track your progress

Project: Build an Image Thumbnail Pipeline with S3, Lambda and DynamoDB (Source Code)

By , trainer LinkedIn Last updated:

Instagram and WhatsApp compress images after upload. You will do the same: S3 upload triggers Lambda, a thumbnail lands in S3, metadata goes to DynamoDB. Region ap-south-1.

What you'll learn in this chapter

  • S3 event trigger to Lambda
  • Write thumb + DynamoDB row
  • Pillow layer or stub fallback
  • Cleanup buckets and function

1. Everyday example

Why do we need this?

A course site uploads a 5 MB photo. The gallery needs a 200 px thumb. Doing that on every page view is wasteful — do it once on upload.

What is it?

ObjectCreated on bucket → Lambda get_object → resize (Pillow layer) or stub copy → put_object under thumbs/ → PutItem metadata.

PipelineUploadS3 originals/EventObjectCreatedLambdaresizeS3thumbs/ + DynamoDB meta
Skip keys that already start with thumbs/.

Download: image-thumbnail-pipeline.zip. Attach a public Pillow layer for real resize, or use the built-in stub path so the lab still verifies.

2. Lab

How to do it (step by step)

A. Buckets + table

  1. Create bucket yourname-imgs (ap-south-1). Block Public Access on.
  2. Optional second bucket for thumbs, or same bucket with prefix thumbs/.
  3. DynamoDB yourname-images, PK imageId String.

B. Lambda

  1. Role: logs + s3:GetObject/PutObject on your buckets + PutItem.
  2. Function Python 3.12, handler.py. Env TABLE_NAME, optional DEST_BUCKET, THUMB_PREFIX=thumbs/.
  3. Optional: Layers → add Pillow community layer for Python 3.12 in ap-south-1.
  4. Timeout 30 s, memory 512 MB.

C. S3 trigger + test

  1. S3 → Properties → Event notifications → All object create → Lambda function. Prefix uploads/ optional.
  2. Upload a small JPG to uploads/demo.jpg.
  3. Check thumbs/ object and DynamoDB item (imageId, thumbKey, bytes).

Cleanup

  1. Remove event notification. Delete Lambda. Empty and delete bucket(s). Delete table and role.

3. Interview

Q. Why not resize in the browser only?

Clients lie; you need a trusted server-side thumb for cards and emails.

Q. Why skip thumbs/ prefix?

Avoid recursive events if thumb lands in the same bucket.

Q. Pillow missing?

Lab stubs a copy so wiring still works; production needs a layer or container image.

Q. Large images?

Raise memory/timeout; consider Step Functions for huge files.

Q. Cleanup?

Notification, Lambda, buckets, table.

Try at home

  1. Write thumbs to a separate bucket with OAC later for CloudFront.
  2. Store contentType in DynamoDB.

In short

  1. S3 event → Lambda → thumb + DynamoDB.
  2. Guard against recursive prefixes.
  3. Layers add native libs like Pillow.
  4. Clean up buckets.

Practice Tasks

About this scenario

Kolhapuri Chappal Co is a footwear business in Kolhapur. Sachin Mane looks after its IT and cloud work. They count money in INR and run their AWS resources in the ap-south-1 Region. In the tasks below you do real jobs for them using what this lesson teaches. All names, account IDs, IP addresses and numbers are made up for practice.

0 / 12

Easy List functions

Why: See which functions exist in this Region.

What to do: List the Lambda functions in ap-south-1.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Medium Invoke a function

Why: Invoke runs the function now and saves its answer in a file.

What to do: Invoke kolhapurichappal-orders-fn and save the response to out.json.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Medium Trust policy for a Lambda role

Why: The trust policy says WHO may take the role.

What to do: Write a trust policy that lets lambda.amazonaws.com assume the role.

How:
  1. Start with { "Version": "2012-10-17", "Statement": [ ... ] }.
  2. In each statement write Effect, Action and Resource.
  3. Give only what the task needs (least privilege).
  4. Use double quotes and commas correctly. Click Check answer.

Medium Put an item

Why: DynamoDB JSON shows the type of each value: S for string, N for number.

What to do: Put an item into kolhapurichappal-orders with orderId "o-101" (string) and amount 2500 (number).

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Medium Change timeout and memory

Why: More memory also gives more CPU; timeout stops runaway runs.

What to do: Set kolhapurichappal-orders-fn to timeout 30 seconds and memory 512 MB.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Medium Let a function write logs

Why: Without these actions, Lambda logs never appear in CloudWatch.

What to do: Allow logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents on arn:aws:logs:*:*:* only.

How:
  1. Start with { "Version": "2012-10-17", "Statement": [ ... ] }.
  2. In each statement write Effect, Action and Resource.
  3. Give only what the task needs (least privilege).
  4. Use double quotes and commas correctly. Click Check answer.

Hard Create a Lambda function

Why: Lambda needs the code zip, a runtime, a handler name and an execution role.

What to do: Create function kolhapurichappal-orders-fn: runtime python3.12, handler app.handler, role arn:aws:iam::118690042069:role/lambda-basic, code in function.zip.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Hard Set environment variables

Why: Keep settings like table names out of the code.

What to do: Set the environment variable TABLE=kolhapurichappal-orders on kolhapurichappal-orders-fn.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Hard Invoke with a JSON payload

Why: In CLI v2, raw JSON needs --cli-binary-format raw-in-base64-out.

What to do: Invoke kolhapurichappal-orders-fn with payload {"orderId":"o-101"} (raw JSON) and save the reply in out.json.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Hard Lambda GB-seconds

Why: Lambda bills memory × time.

What to do: 1,000,000 invocations, 512 MB memory, 200 ms each. How many GB-seconds?

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Connect an SQS queue to Lambda

Why: An event source mapping makes Lambda poll the queue and process messages in batches.

What to do: Create an event source mapping from queue arn:aws:sqs:ap-south-1:118690042069:kolhapurichappal-orders to kolhapurichappal-orders-fn with batch size 10.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Hard Let S3 call your function

Why: A resource-based permission is needed before S3 can invoke the function.

What to do: Add permission statement s3invoke to kolhapurichappal-orders-fn so s3.amazonaws.com can call lambda:InvokeFunction from bucket arn:aws:s3:::kolhapurichappal-uploads.

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

See all practice tasks in the Task Hub →

Revision, comparison and interview cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

Revision card: Image thumbnail pipeline. A 5 MB course photo should not download full size for every gallery card. You need a small thumb once. Upload to S3 → ObjectCreated invokes Lambda → resize → put thumb under thumbs/ → PutItem metadata in DynamoDB. Bucket yourname-imgs, table PK imageId, Lambda with S3 Get/Put + PutItem, S3 event on uploads/ prefix, optional Pillow layer. Upload uploads/demo.jpg → object thumbs/demo.jpg appears and DynamoDB has imageId, thumbKey, bytes.
Revision card: Image thumbnail pipeline · Download PNG · Open card page
Revision card: Avoid recursive S3 events. If Lambda writes a thumb into the same prefix it listens on, every write fires another invoke forever. Guard: only process keys under uploads/ (or skip any key starting with thumbs/). Write thumbs elsewhere. Event notification prefix uploads/. In code: if key.startswith("thumbs/"): return. Dest prefix THUMB_PREFIX=thumbs/. One upload creates exactly one thumb and one DynamoDB row — CloudWatch shows a single invoke.
Revision card: Avoid recursive S3 events · Download PNG · Open card page
Revision card: Pillow layer vs stub. Python zip uploads cannot easily ship native image libraries. Labs still need a wiring test. A Lambda layer can attach Pillow for real resize. Without it, a stub copies the object so S3+DynamoDB wiring still verifies. Layers › Add a community Pillow layer for Python 3.12 in ap-south-1, or leave stub path on. Memory 512 MB, timeout 30 s. With Pillow, a 2000 px JPG becomes ~200 px wide under thumbs/; without Pillow, thumb bytes ≈ original but flow works.
Revision card: Pillow layer vs stub · Download PNG · Open card page
Comparison: S3 ObjectCreated trigger vs API Gateway HTTP API. S3 ObjectCreated trigger: Upload creates an event automatically. API Gateway HTTP API: Client must call an HTTPS route.
Comparison: S3 ObjectCreated trigger vs API Gateway HTTP API · Download PNG · Open card page
Comparison: Same bucket + thumbs/ prefix vs Separate thumb bucket. Same bucket + thumbs/ prefix: Must skip thumbs/ in code or event prefix. Separate thumb bucket: Natural isolation of originals vs thumbs.
Comparison: Same bucket + thumbs/ prefix vs Separate thumb bucket · Download PNG · Open card page
Interview card: Why skip keys under thumbs/?. Writing the thumb into a watched prefix would invoke Lambda again (recursion). Bills and throttles explode until you delete the notification. Listen on uploads/ only, or if key.startswith("thumbs/"): return before resize.
Interview card: Why skip keys under thumbs/? · Download PNG · Open card page
Interview card: How do you add Pillow to Lambda?. Attach a Lambda layer that contains the Pillow wheel for your runtime and architecture. Zip size limits and native binaries make layers the usual path (or a container image). Function › Layers › Add layer › community Pillow for Python 3.12 x86_64 in ap-south-1.
Interview card: How do you add Pillow to Lambda? · Download PNG · Open card page

More cards: Ravi Sir's Cards.