Ravindra BagaleCourses & study guides मराठी Track your progress

DynamoDB Streams with Lambda: Change Data Capture Patterns

By , trainer LinkedIn Last updated:

CRUD mentioned Streams briefly. Toronto ops lead Jordan Lee lost audit events because Lambda errors retried forever on a poison record without a DLQ. This chapter: stream view types, Lambda CDC, shards, fan-out, exactly-once pitfalls. Link: CRUD Streams note.

What you'll learn in this chapter

  • Explain with an everyday story
  • Apply Why/What/How
  • Read commands with output+meaning
  • Map across industries
  • Finish lab + cleanup

1. Everyday example: CCTV tape of the shop till

Every time the till drawer opens (item change), a camera writes a clip (stream record). Guards (Lambda) watch clips. KEYS_ONLY is a note of which drawer; NEW_AND_OLD is before/after video — heavier. If one clip is corrupt, do not block the whole night — send to a dead-letter shelf (DLQ).

2. Vertical steps

Why -> What -> How -> Lab -> CleanupWhyWhatHowLabCleanup
Cleanup labs

3. Why do we need this?

Why do we need this?

Other systems need change events: search index, analytics, audit, cache invalidate. Polling Scans is wasteful and slow.

4. What is it?

What is it?

DynamoDB Streams is an ordered change log per shard. View types control payload size. Lambda event source mapping reads shards. Design for at-least-once delivery with idempotent consumers and a DLQ.

5. How to do it (step by step)

How to do it (step by step)

  1. Enable stream with needed view type only.
  2. Attach Lambda with batch size and DLQ.
  3. Make handler idempotent (use eventID / keys).
  4. Fan-out via SNS/EventBridge if many consumers.
  5. Alarm on iterator age and DLQ depth.

6. Event source mapping notes

Batch size, parallelization factor, starting position TRIM_HORIZON vs LATEST, retry attempts, DLQ ARN. Start LATEST for labs; TRIM_HORIZON for full replay carefully.

7. Fan-out patterns

Lambda publishes to SNS/EventBridge; or Pipeline: Stream -> Lambda -> SQS per consumer. Avoid two heavy Lambdas fighting one shard without need.

8. Commands with output and meaning

aws dynamodb update-table --table-name lab-dda-shop \
  --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES
# Output: StreamSpecification enabled. Meaning: each change includes before/after images.
aws dynamodb describe-table --table-name lab-dda-shop --query 'Table.LatestStreamArn'
# Output: arn:aws:dynamodb:ap-south-1:.../stream/.... Meaning: use this ARN on Lambda event source mapping.

Enabling NEW_AND_OLD_IMAGES gives rich CDC at higher storage/payload cost. LatestStreamArn wires Lambda. Prefer KEYS_ONLY if consumers only need to GetItem.

9. Industry stories

  • Fintech audit (Elena Petrova): NEW_AND_OLD to audit table.
  • Search (Budi Santoso): stream -> OpenSearch indexer.
  • Cache (Minji Park): KEYS_ONLY invalidate DAX/Redis.
  • Analytics (Nour Hassan): stream -> Firehose -> S3.
  • Support (Aisha Khan): stream -> Slack notifier filtered.

10. Lab

How to do it (step by step)

  1. Enable stream on lab table; note ARN.
  2. Optional: attach a Lambda (or document the mapping).
  3. Put/Update an item; confirm stream metrics.
  4. Disable stream; delete table.

Cleanup (stop billing)

  1. Delete lab tables/indexes/triggers.
  2. Confirm describe-table fails or status missing.

11. Access-pattern workshop (paper first)

  1. List every screen or API that reads or writes data.
  2. For each: write the exact key lookup or query shape in plain English.
  3. Mark 1:1, 1:many, many:many relationships.
  4. Circle anything that needs a sort range (time, status, score).
  5. Only then invent PK/SK and optional GSI keys.
  6. Estimate peak items per partition key per second.
  7. Write one sample item JSON for each entity type.
  8. Peer-review with someone who did not design it.

12. Copy-paste DynamoDB CLI starters

# Region habit for this series
export AWS_DEFAULT_REGION=ap-south-1

# Describe + sample
aws dynamodb describe-table --table-name ShopApp
aws dynamodb scan --table-name ShopApp --max-items 3

# Query by PK (replace names)
aws dynamodb query --table-name ShopApp   --key-condition-expression "PK = :pk"   --expression-attribute-values '{":pk":{"S":"USER#u123"}}'

# On-demand capacity mode check
aws dynamodb describe-table --table-name ShopApp   --query 'Table.BillingModeSummary'

# Streams status
aws dynamodb describe-table --table-name ShopApp   --query 'Table.StreamSpecification'

13. More industry stories

  • Retail (Aisha Khan, Dubai): single-table cart+orders; PK=USER#id keeps each shopper hot path local.
  • Gaming (Carlos Rivera, Mexico City): leaderboard uses sparse GSI on score; write shard on hot gameId.
  • IoT (Mei Lin, Shenzhen): sensor#day PK + timestamp SK; TTL 30 days keeps storage flat.
  • Healthcare (Rohan Deshmukh, Pune): appointment slots use conditional Put — no double book.
  • Fintech (Elena Petrova, Warsaw): Streams fan-out to audit table; never update audit in place.
  • Travel (Omar Haddad, Amman): global tables for read-local catalog; writes region-sticky for inventory.

14. Glossary (short)

TermMeaning
PK / Partition keyHash key that places the item in a partition
SK / Sort keyOrders items inside a partition; enables ranges
Item collectionAll items that share the same PK
GSIGlobal Secondary Index — alternate PK/SK, own partitions
LSILocal Secondary Index — same PK, different SK
RCU / WCURead/Write Capacity Units on provisioned mode
Conditional writeWrite succeeds only if a condition is true
Stream view typeKEYS_ONLY / NEW_IMAGE / OLD_IMAGE / NEW_AND_OLD_IMAGES

15. Practice: explain out loud

  1. 60s: why access patterns come before table design.
  2. 60s: hot partition and one fix (write sharding).
  3. 60s: GSI vs LSI in one breath.
  4. 60s: on-demand vs provisioned for a startup.
  5. 60s: optimistic locking with a version attribute.
  6. 60s: Streams + Lambda for an audit trail.
  7. Record once; cut filler; record again.

16. Safe lab hygiene

  • Use ap-south-1 labs with a budget alert (₹/USD).
  • Prefix lab tables with lab- and your initials.
  • Prefer on-demand for tiny labs unless you are learning auto scaling.
  • Delete tables and Lambda triggers after the lab.
  • Never point Streams consumers at production without a DLQ.
  • Tag every resource Owner + Expiry.

18. Single-table sketch pad

Write entities as rows in one mental table before you open the console. Example for a small shop: USER#, ORD#, ORD#|ITEM#, PROD# under carefully chosen PK/SK.

EntityPKSKExample attrs
UserUSER#u1PROFILEname, city
OrderUSER#u1ORD#2026-10-31#o9total, status
Order itemORD#o9ITEM#sku1qty, price
ProductPROD#sku1METAtitle, stock

19. Ten oral drill questions

  1. What is an access pattern?
  2. Why overload PK with entity prefixes?
  3. Name one hot-partition symptom.
  4. Sparse GSI means what?
  5. When is LSI impossible after create?
  6. On-demand bills for what?
  7. What does attribute_not_exists do?
  8. NEW_AND_OLD_IMAGES costs more — why still pick it?
  9. How do you stop a poison Streams message loop?
  10. Which existing chapter do you open for PutItem syntax?

20. Capacity and throttle drill

  1. Open CloudWatch for your table: ConsumedReadCapacityUnits, ConsumedWriteCapacityUnits, ThrottledRequests.
  2. Note whether BillingMode is PAY_PER_REQUEST or PROVISIONED.
  3. If provisioned, write current RCU/WCU and autoscaling min/max.
  4. Name your hottest expected PK in one sentence.
  5. Write one mitigation: shard, cache, or redesign.
  6. Set or verify a budget alert for DynamoDB.
  7. Check GSI list and whether each is still used.
  8. Confirm Streams enabled only where consumers exist.

21. Condition expression cheatsheet

# Create-once
attribute_not_exists(PK)

# Optimistic lock
version = :v

# Stock / seat
stock > :zero
#status = :free

# Combine
attribute_not_exists(PK) AND #type = :user

Meaning: conditions are the seatbelt for concurrent writers. Put them on every race-prone update. Full SDK samples stay in the SDK chapter — here we only list the expressions you will reuse.

22. Streams consumer checklist

  • View type matches need (KEYS_ONLY vs NEW_AND_OLD).
  • Lambda has DLQ and retry policy.
  • Handler idempotent on eventID or keys.
  • IteratorAge alarm before users feel lag.
  • No accidental write-loop back into the same table without markers.
  • Fan-out via SNS/EventBridge when >1 consumer team.
  • Disable stream when lab ends.
  • Ch1 Modeling -> Ch2 Hot keys -> Ch3 GSI/LSI -> Ch4 Capacity -> Ch5 Transactions -> Ch6 Streams.
  • Always link intro + CRUD + CLI + SDK instead of rewriting Put/Get.
  • Projects P1-P7 come in later batches and reuse these ideas.

24. Extra oral drills

  1. Explain single-table to a product manager in 45 seconds.
  2. Draw PK/SK for a chat thread on a whiteboard.
  3. Name three CloudWatch metrics for DynamoDB pain.
  4. Say when you would refuse a new GSI request.
  5. Describe one transaction for seat + payment record.
  6. Explain why Streams are at-least-once.
  7. Convert a Scan-based feature request into a Query design.
  8. List cleanup steps after a DynamoDB lab.

25. Teacher story: the notebook that became a database

Ravindra once watched a shop keep every customer, order and payment in one paper notebook. Finding last Tuesday's unpaid bills meant flipping pages. DynamoDB single-table thinking is that notebook done carefully: one place, clear labels (PK/SK), and an index tab (GSI) only when a second way to find rows is worth the cost. Do not photocopy the notebook into ten unrelated books unless the shop truly has ten separate businesses.

When Meera (clinic), Omar (travel), and Aisha (retail) design together, they still start on paper: screen list, key sketch, sample JSON, peer review. Console clicks come last. That ritual is why this deep series links the intro and CRUD chapters instead of rewriting PutItem on every page.

26. Sixteen mini tasks (self-check, leak-free)

  1. Write three access patterns for a cart app in one minute.
  2. Invent PK/SK for user profile and orders under that user.
  3. Name one hot-partition anti-pattern.
  4. Explain write sharding to a teammate in 40 seconds.
  5. Say GSI vs LSI without notes.
  6. Pick KEYS_ONLY or ALL for a sparse open-orders index — why?
  7. Choose on-demand or provisioned for a weekend flash sale.
  8. List five DynamoDB bill line-item types.
  9. Write a create-once condition in words.
  10. Describe optimistic locking with a version attribute.
  11. When do you wrap two writes in a transaction?
  12. Pick a stream view type for an audit log — justify.
  13. Name two Stream consumer safety rules.
  14. What CloudWatch metric shows stream lag?
  15. Which linked chapter has PutItem CLI syntax?
  16. Write your lab cleanup checklist (table, stream, lambda, alarms).

27. Marathi classroom lines (say out loud)

  • Access patterns आधी, keys नंतर — हे mantra आठवा.
  • Hot partition म्हणजे एक PK वर गर्दी — shard किंवा redesign.
  • GSI लवचिक; LSI create-time — चुकीचा index महाग.
  • On-demand म्हणजे वापरल्यावर पैसे; provisioned म्हणजे आरक्षित units.
  • Condition आणि transaction म्हणजे race साठी seatbelt.
  • Streams at-least-once आहे — handler idempotent ठेवा.

28. Hindi bridge lines

  • Pehle access patterns likho, baad mein PK/SK.
  • Hot partition ek key par zyada load.
  • GSI kabhi bhi; LSI table banate waqt.
  • On-demand spike ke liye; provisioned steady ke liye.
  • Transaction = sab milkar ya kuch nahi.
  • Streams duplicate de sakte hain — idempotent likho.

29. Common errors and fixes

Each row is a real mistake students and juniors hit. Read the fix before you panic.

Error / symptomFix
No DLQ on stream LambdaAdd SQS DLQ.
Non-idempotent handlerKey on eventID/keys.
NEW_AND_OLD alwaysUse KEYS_ONLY if enough.
Blocking poison recordBisect batch / DLQ.
Multiple Lambdas on one stream without fan-out planUse ESM carefully or SNS fan-out.
Ignoring IteratorAge alarmAlarm iterator age.
Updating base inside same stream blindlyAvoid loops with markers.
Assuming exactly-onceDesign at-least-once.
Forgetting shard parallelism limitsWatch concurrency.

30. Troubleshooting checklist

  1. No events: stream disabled or wrong table.
  2. IteratorAge rising: Lambda errors/throttle.
  3. Duplicates in target: missing idempotency.
  4. Looping writes: filter self-written attrs.
  5. Huge payloads: switch view type.

31. FAQ

Q. Should I always use a single table?

Often yes for related access patterns; not a religion. Separate tables are fine for unrelated domains or different capacity/security needs.

Q. Do I still need RDS?

Yes for heavy joins, multi-row SQL reports, and strong relational constraints. See the intro DynamoDB vs RDS chapter.

Q. Is Scan evil?

Scan is a tool for rare admin jobs and small tables. Design so hot paths use Query/GetItem.

Q. Where are Put/Get command details?

In the CRUD survey, CLI and SDK chapters — this deep set links them instead of copying.

Q. Stream view types?

KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, NEW_AND_OLD_IMAGES.

Q. At-least-once?

Records may replay; consumers must be idempotent.

Q. DLQ?

Dead-letter queue for failed batches.

Q. Fan-out?

One stream to many consumers via messaging.

32. Interview Q&A

Q. (B) Why Streams?

CDC without polling Scans.

Q. (B) KEYS_ONLY use?

Light notify then GetItem.

Q. (M) Poison message?

Fails forever — DLQ + fix.

Q. (M) Exactly-once?

Not guaranteed — idempotent design.

Q. (S/MNC) Audit architecture?

NEW_AND_OLD -> append-only audit store; no in-place edit.

Q. (MNC) IteratorAge SLO?

Alarm before user-visible lag.

33. Tips in English, Marathi and Hindi

Ravindra Bagale's Tip (EN)

Streams are at-least-once. Idempotent consumers + DLQ are not optional.

Ravindra Bagale's Tip (MR)

At-least-once; idempotent + DLQ compulsory.

Tip (HI)

Stream = at-least-once। Idempotent + DLQ जरूरी।

Try at home

  1. Pick a view type for an audit use case.
  2. Write an idempotency rule for your handler.
  3. Re-read CRUD Streams paragraph.

In short

  1. Streams = ordered CDC.
  2. Pick lean view types.
  3. Lambda + DLQ + idempotent.
  4. Fan-out when many consumers.

Revision, interview and comparison cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

Revision card: Streams CDC map. Polling the table for changes is slow and expensive. Enable Streams → pick view type → Lambda (idempotent + DLQ) → sink. IteratorAge alarm → fan-out via SNS/EventBridge when many teams. Fintech audit for Elena uses NEW_AND_OLD_IMAGES into an append-only table.
Revision card: Streams CDC map · Download PNG · Open card page
Revision card: Safe stream consumers. Poison events and write-loops stall shards. Idempotent on eventID/keys, DLQ, no unmarked write-back loops. Build handler → DLQ → alarm → prove replay → disable in lab. Kenji Watanabe refuses production Streams without a DLQ ticket.
Revision card: Safe stream consumers · Download PNG · Open card page
Comparison: KEYS_ONLY vs NEW_AND_OLD_IMAGES. KEYS_ONLY: Fetch if needed. NEW_AND_OLD_IMAGES: Both images in record.
Comparison: KEYS_ONLY vs NEW_AND_OLD_IMAGES · Download PNG · Open card page
Comparison: Direct many Lambdas vs SNS/EventBridge fan-out. Direct many Lambdas: Every team attaches to stream. SNS/EventBridge fan-out: One consumer publishes events.
Comparison: Direct many Lambdas vs SNS/EventBridge fan-out · Download PNG · Open card page
Interview card: Which stream view type?. Choice. KEYS_ONLY vs NEW/OLD/NEW_AND_OLD — pick for sink needs vs cost. Audit often wants both images.
Interview card: Which stream view type? · Download PNG · Open card page
Interview card: Why idempotent consumers?. Delivery. Streams are at-least-once; duplicates happen. Key on eventID or business keys.
Interview card: Why idempotent consumers? · Download PNG · Open card page
Interview card: IteratorAge is rising — what?. Ops. Consumer lag; scale, fix errors, check DLQ/poison. Alarm before users feel it.
Interview card: IteratorAge is rising — what? · Download PNG · Open card page
Interview card: How to fan-out CDC?. Architecture. One stream consumer publishes to SNS/EventBridge for many teams. Avoid 20 Lambdas on one stream blindly.
Interview card: How to fan-out CDC? · Download PNG · Open card page

More cards: Ravi Sir's Cards.

Practice Tasks

About this scenario

Ahmednagar Sugar Co-op is a sugar co-operative business in Ahilyanagar. Balasaheb Pawar looks after its IT and cloud work. They count money in INR and run their AWS resources in the ap-south-1 Region. In the tasks below you do real jobs for them using what this lesson teaches. All names, account IDs, IP addresses and numbers are made up for practice.

0 / 16

Easy Streams capture item changes — 1

Why: Def.

What to do: Enter 1 if DynamoDB Streams capture item-level table changes.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy NEW_AND_OLD has both images — 1

Why: View.

What to do: Enter 1 if NEW_AND_OLD_IMAGES includes both new and old item images.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy At-least-once needs idempotent handlers — 1

Why: Delivery.

What to do: Enter 1 if stream delivery is at-least-once so handlers must be idempotent.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy IteratorAge shows lag — 1

Why: Metric.

What to do: Enter 1 if IteratorAge rising means consumer lag.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy Describe table stream spec

Why: CLI.

What to do: Describe table ShopApp

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Easy View type YAML

Why: Pick.

What to do: YAML: audit: NEW_AND_OLD_IMAGES, keys_only_ok: KEYS_ONLY.

How:
  1. Start from the top-level keys the task names.
  2. Indent child keys with 2 spaces (no tabs).
  3. Fill in each property with the value from the task.
  4. Click Check answer. Key order does not matter.

Easy Enable Streams before consumers — 1

Why: Order.

What to do: Enter 1 if the table stream must be enabled before consumers read it.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium DLQ for poison events — 1

Why: Safety.

What to do: Enter 1 if a DLQ helps isolate poison stream events.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Fan-out via SNS/EventBridge — 1

Why: Pattern.

What to do: Enter 1 if many consumer teams often fan-out via SNS/EventBridge.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium KEYS_ONLY then GetItem — 1

Why: View.

What to do: Enter 1 if KEYS_ONLY can be enough when you GetItem as needed.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Avoid write loops — 1

Why: Risk.

What to do: Enter 1 if consumers writing back can create endless stream loops.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Consumer checklist YAML

Why: List.

What to do: YAML list checks with dlq and idempotent.

How:
  1. Start from the top-level keys the task names.
  2. Indent child keys with 2 spaces (no tabs).
  3. Fill in each property with the value from the task.
  4. Click Check answer. Key order does not matter.

Hard CRUD has a short Streams note — 1

Why: Complement.

What to do: Enter 1 if the CRUD chapter has a short Streams note this deepens.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Disable Streams after lab — 1

Why: Hygiene.

What to do: Enter 1 if you should disable Streams when lab consumers are gone.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard CDC can feed OpenSearch/S3 — 1

Why: Sink.

What to do: Enter 1 if CDC patterns often fan out to OpenSearch or S3.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Shards parallelize stream processing — 1

Why: Parallel.

What to do: Enter 1 if stream shards allow parallel consumers within limits.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

See all practice tasks in the Task Hub →