Ravindra BagaleCourses & study guides मराठी Track your progress

Project: CDC Streams to Lambda to OpenSearch

By , trainer LinkedIn Last updated:

Users search text; DynamoDB does not. Austin search lead Maya Brooks and Pune founder Vikram Patil ship FindLite: Streams → Lambda → OpenSearch upsert/delete, IteratorAge alarms, DLQ, S3 backfill. Reuse Streams; do not rewrite Put/Get.

What you'll learn in this chapter

  • Enable NEW_AND_OLD stream
  • Idempotent upsert/delete in Lambda
  • Alarm IteratorAge + DLQ
  • Backfill from export
  • Lab + cleanup

1. Everyday example: library card catalog

Shelves are system of record (DynamoDB). Card catalog (OpenSearch) updates when a book is added/moved/removed. Lagging cards confuse readers — watch IteratorAge.

2. Vertical steps

Patterns -> Stream -> Lambda index -> Alarms/DLQ -> Backfill -> CleanupPatternsStreamLambda indexAlarms/DLQBackfillCleanup
Cleanup

3. Why do we need this?

Why do we need this?

Product search, ticket search, and directories need relevance and fuzzy match. MNCs ask CDC design, exactly-once myths, and poison-message handling.

4. What is it?

What is it?

FindLite: table stream NEW_AND_OLD_IMAGES → Lambda (ReportBatchItemFailures) → OpenSearch index alias. _id=PK|SK. INSERT/MODIFY upsert; REMOVE delete. DLQ + IteratorAge alarm. Backfill: Export→S3→bulk then tail stream.

5. How to do it (step by step)

How to do it (step by step)

  1. List searchable fields; exclude PII you must not index.
  2. Enable stream NEW_AND_OLD on lab-ddq-catalog.
  3. Lambda: map eventName to index/delete; external version.
  4. Put/Modify/Delete items; search via OpenSearch API.
  5. Force poison record → DLQ; alarm IteratorAge.
  6. Paper backfill steps; delete lab resources.

6. Architecture sketch

PieceRoleNote
DynamoDBSystem of recordKeys unchanged
StreamsCDC feedAt-least-once
LambdaTransform+indexIdempotent
OpenSearchSearch UXWrite via alias
DLQ+alarmsPoison+lagIteratorAge

7. Commands with output and meaning

aws dynamodb create-table --table-name lab-ddq-catalog \
  --attribute-definitions AttributeName=PK,AttributeType=S AttributeName=SK,AttributeType=S \
  --key-schema AttributeName=PK,KeyType=HASH AttributeName=SK,KeyType=RANGE \
  --billing-mode PAY_PER_REQUEST \
  --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES \
  --region ap-south-1
aws dynamodb describe-table --table-name lab-ddq-catalog \
  --query 'Table.LatestStreamArn'
aws dynamodb put-item --table-name lab-ddq-catalog --item '{
 "PK":{"S":"PROD#9"},"SK":{"S":"META"},"title":{"S":"Blue mug"},"tags":{"S":"kitchen"}}'
# Wire event source mapping to Lambda (console/SAM/CDK). Meaning: each change becomes a search mutation.

At-least-once ⇒ upsert with version. Always delete on REMOVE. Backfill before relying on tail only.

8. Industry stories

  • E-commerce (Aisha): product catalog search.
  • Support (Lena): ticket full-text.
  • HR (Omar): employee directory.
  • Media (Priya): article search.
  • Legal (Maya): case metadata search.
  • IoT ops (Diego): device inventory search.

9. Lab

How to do it (step by step)

  1. Create lab-ddq-catalog with stream; note StreamArn.
  2. Stub Lambda upsert/delete; put+delete item; confirm search intent on paper if OS unavailable.
  3. Document DLQ + IteratorAge alarm; delete table.

Cleanup (stop billing)

  1. Delete lab resources you created (table, Lambda test, indexes).
  2. Confirm with describe that names are gone.

10. Ops checklist before you ship

  1. Confirm TTL attribute name and units (epoch seconds).
  2. Decide soft-delete vs hard-delete vs archive-to-S3.
  3. Enable PITR on any table you cannot rebuild from source.
  4. Document RPO/RTO and who runs restore drills.
  5. If multi-region: pick write stickiness and conflict rules.
  6. If DAX: prove cache-miss path and warm-up plan.
  7. Tag Owner + Expiry on every lab resource.
  8. Budget alert in the account before load tests.

11. Copy-paste DynamoDB CLI starters (batch 2)

export AWS_DEFAULT_REGION=ap-south-1

# TTL status
aws dynamodb describe-time-to-live --table-name ShopApp

# PITR / continuous backups
aws dynamodb describe-continuous-backups --table-name ShopApp

# On-demand backup
aws dynamodb create-backup --table-name ShopApp --backup-name lab-shop-$(date +%Y%m%d)

# List backups
aws dynamodb list-backups --table-name ShopApp

# Global table description (v2)
aws dynamodb describe-table --table-name ShopApp --query 'Table.Replicas'

12. More industry stories

  • Retail (Aisha Khan, Dubai): TTL clears abandoned carts; soft-delete keeps 30-day undo.
  • Bank (Lucas Meyer, Berlin): PITR on ledger table; restore drill every quarter.
  • Travel (Omar Haddad, Amman): Global Tables for catalog reads; inventory writes sticky.
  • Gaming (Carlos Rivera, Mexico City): DAX for hot profile Gets; never cache money ledgers.
  • Social (Priya Nair, Kochi): adjacency edges for follows; sparse GSI for unread inbox.
  • IoT (Mei Lin, Shenzhen): device#day PK + TTL 14 days; rollups to S3.

13. Glossary (batch 2)

TermMeaning
TTLTime to Live — attribute with epoch seconds; DynamoDB deletes later
Soft deleteFlag deleted=true; item remains until TTL/archive
PITRPoint-in-time recovery — continuous backups window
Global TableMulti-region replicated DynamoDB table
LWWLast-writer-wins conflict resolution on replicas
DAXDynamoDB Accelerator — in-memory cache cluster
Adjacency listEdge items modeling relationships in one table
Fan-outWrite once; push copies to many feeds/inboxes

14. Practice: explain out loud

  1. 60s: TTL vs soft-delete vs S3 archive.
  2. 60s: PITR vs on-demand backup.
  3. 60s: when Global Tables help and hurt.
  4. 60s: DAX consistency trade-off.
  5. 60s: CartLite checkout keys in one breath.
  6. 60s: follow edge vs feed item.
  7. Record once; cut filler; record again.

15. Safe lab hygiene

  • Use ap-south-1 labs with a budget alert.
  • Prefix lab tables with lab- and your initials.
  • Delete replicas, DAX clusters, and backups you created for labs.
  • Never enable Global Tables on production without a write plan.
  • Tag every resource Owner + Expiry.
  • Export/import labs can create large S3 bills — delete prefixes.

17. Delete / archive decision pad

NeedPreferWhy
Undo 7-30 daysSoft-delete flagUser restore
Hard expiry, no readTTL attributeAuto delete later
Compliance keep coldExport/archive to S3Cheap long store
Mistake recoveryPITR / backupTable-level restore

18. Ten oral drill questions

  1. What units does DynamoDB TTL use?
  2. Does TTL delete instantly?
  3. PITR window typical length?
  4. Global Tables conflict default?
  5. When sticky writes matter?
  6. DAX vs API Gateway cache?
  7. Cart PK for a shopper?
  8. Order lines under which PK?
  9. Follow edge SK example?
  10. Unread inbox sparse GSI idea?

19. RPO / RTO mini workshop

  1. Write the worst data loss you can accept (RPO) in minutes.
  2. Write how long restore may take (RTO) before business pain.
  3. Map: on-demand backup vs PITR vs multi-region replica.
  4. Schedule a restore drill on a lab table this month.
  5. Name the person who runs the drill.

20. Cost watch list (batch 2)

  • PITR storage and on-demand backup storage.
  • Global Tables: replicated WCUs in every region.
  • DAX node hours even at idle.
  • Export to S3 + S3 storage + Athena later.
  • Streams + Lambda still bill during TTL deletes.

21. Project Definition of Done

  1. Access patterns written and peer-reviewed.
  2. Sample items drawn for happy path + one failure.
  3. Lab table created, exercised, deleted (or tagged Expiry).
  4. At least one conditional write or transaction exercised.
  5. Cleanup commands pasted in your notes.
  6. One interview answer recorded out loud.
  • TTL hygiene → Backup/PITR for “oops” recovery.
  • Global Tables → sticky writes from capacity/hot lessons.
  • DAX → only after Query shapes are correct.
  • P1 CartLite reuses modeling + transactions.
  • P2 Social reuses adjacency + Streams ideas.

23. Quick self-check (yes/no)

  1. I can explain TTL delay vs instant DeleteItem.
  2. I know when soft-delete beats hard delete.
  3. I can start PITR and say what restore costs in time.
  4. I can name one reason NOT to use Global Tables.
  5. I know DAX does not fix a bad partition key.
  6. I can sketch CartLite USER# and ORD# keys.
  7. I can sketch a follow edge item.
  8. I delete lab replicas/DAX/backups the same day.

24. Common errors and fixes

Each row is a real mistake students and juniors hit. Read the fix before you panic.

Error / symptomFix
TrimmedDataAccessExceptionConsumer too slow — scale/fix errors.
Assuming exactly-onceIdempotent upsert+version.
Ignore REMOVEDelete-by-id in OpenSearch.
No DLQPoison blocks shard.
Silent lagAlarm IteratorAge.
Index PII blindlyField allow-list.
Backfill skippedExport→bulk then tail.
Leave lab resourcesDelete same day.
Rewrite Streams chapterLink Streams.

25. Troubleshooting checklist

  1. IteratorAge rising: errors, throttle, or under-provisioned Lambda.
  2. Version conflicts: use stream sequence / timestamp external version.
  3. Mapping errors: fix index template before replay.
  4. Partial batch: ReportBatchItemFailures.
  5. Count mismatch: reconcile job by PK prefix.

26. FAQ

Q. Should I always use a single table?

Often yes for related access patterns; not a religion. Separate tables are fine for unrelated domains or different capacity/security needs.

Q. Do I still need RDS?

Yes for heavy joins, multi-row SQL reports, and strong relational constraints. See the intro DynamoDB vs RDS chapter.

Q. Is Scan evil?

Scan is a tool for rare admin jobs and small tables. Design so hot paths use Query/GetItem.

Q. Where are Put/Get command details?

In the CRUD survey, CLI and SDK chapters — this deep set links them instead of copying.

Q. OpenSearch Serverless vs managed?

Either; IAM and VPC differ — pick with ops.

Q. Pipe without Lambda?

OpenSearch Ingestion / EventBridge pipes possible.

Q. Exactly-once?

Practical answer = idempotent at-least-once.

Q. Huge attrs?

Strip; S3 pointer in doc.

27. Interview Q&A

Q. (B) Why OpenSearch?

Full-text/relevance DynamoDB lacks.

Q. (B) Stream view?

NEW_AND_OLD_IMAGES typical.

Q. (M) Idempotency?

Upsert with external version; delete on REMOVE.

Q. (M) Poison message?

DLQ + alarm; skip/fix.

Q. (S/MNC) Backfill?

Export S3 bulk then enable tail.

Q. (MNC) Exactly-once claim?

At-least-once + idempotent consumer.

28. Tips in English, Marathi and Hindi

Ravindra Bagale's Tip (EN)

CDC is at-least-once — make the consumer boring and idempotent.

Ravindra Bagale's Tip (MR)

CDC = at-least-once; consumer idempotent ठेवा.

Tip (HI)

CDC कम-से-कम एक बार — consumer idempotent रखें।

Try at home

  1. List fields to index vs exclude.
  2. Draw Streams→Lambda→OS + DLQ.
  3. Write IteratorAge alarm threshold.

In short

  1. FindLite = Streams→Lambda→OpenSearch.
  2. Upsert + REMOVE delete.
  3. IteratorAge + DLQ + backfill.
  4. Cleanup labs.

Revision, interview and comparison cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

Revision card: CDC to search. DynamoDB Query is not full-text search. Streams → Lambda → OpenSearch upsert/delete; _id=PK|SK. Enable stream → wire Lambda → Put/Modify/Delete → verify search. Maya Brooks treats DynamoDB as SoR and OS as UX search.
Revision card: CDC to search · Download PNG · Open card page
Revision card: Idempotent consumer. Exactly-once myths hide poison and lag. At-least-once + versioned upsert; DLQ; IteratorAge alarm. ReportBatchItemFailures → DLQ → alarm age → backfill Export then tail. Vikram Patil rehearses poison replay before launch.
Revision card: Idempotent consumer · Download PNG · Open card page
Comparison: DynamoDB Query only vs OpenSearch CDC. DynamoDB Query only: Query/begins_with for product search UX. OpenSearch CDC: Streams→Lambda→OpenSearch full-text.
Comparison: DynamoDB Query only vs OpenSearch CDC · Download PNG · Open card page
Comparison: Exactly-once claim vs At-least-once + idempotent. Exactly-once claim: Assume the stream is exactly once. At-least-once + idempotent: Idempotent upsert with external version.
Comparison: Exactly-once claim vs At-least-once + idempotent · Download PNG · Open card page
Interview card: Enable stream. No stream means no CDC. StreamEnabled + NEW_AND_OLD_IMAGES; note StreamArn. Create table with stream → describe LatestStreamArn → wire Lambda.
Interview card: Enable stream · Download PNG · Open card page
Interview card: Handle REMOVE. Ignoring REMOVE leaves ghost search hits. On REMOVE delete-by-id in OpenSearch (_id=PK|SK). Put doc → DeleteItem → confirm search gone.
Interview card: Handle REMOVE · Download PNG · Open card page
Interview card: Poison DLQ. One bad record can stall a shard. ReportBatchItemFailures + DLQ + alarm. Inject bad mapping → land DLQ → fix template → replay.
Interview card: Poison DLQ · Download PNG · Open card page
Interview card: Export backfill. Tail-only misses history before enable. Export to S3 → bulk index → then enable/tail stream. Run export → bulk → compare counts → open stream.
Interview card: Export backfill · Download PNG · Open card page

More cards: Ravi Sir's Cards.

Practice Tasks

About this scenario

Milan Furniture is a furniture business in Milan. Giulia Rossi looks after its IT and cloud work. They count money in EUR and run their AWS resources in the eu-south-1 Region. In the tasks below you do real jobs for them using what this lesson teaches. All names, account IDs, IP addresses and numbers are made up for practice.

0 / 16

Easy OpenSearch for full-text — 1

Why: Search.

What to do: Enter 1 if OpenSearch adds full-text DynamoDB lacks.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy NEW_AND_OLD_IMAGES — 1

Why: Stream.

What to do: Enter 1 if CDC often uses NEW_AND_OLD_IMAGES.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy At-least-once delivery — 1

Why: CDC.

What to do: Enter 1 if Streams are at-least-once.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy REMOVE deletes search doc — 1

Why: Sync.

What to do: Enter 1 if REMOVE should delete the OpenSearch doc.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy Describe catalog table

Why: CLI.

What to do: Describe table ShopApp

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Easy Pipeline YAML

Why: Arch.

What to do: YAML: source: streams, sink: opensearch.

How:
  1. Start from the top-level keys the task names.
  2. Indent child keys with 2 spaces (no tabs).
  3. Fill in each property with the value from the task.
  4. Click Check answer. Key order does not matter.

Easy Doc _id from PK|SK — 1

Why: Map.

What to do: Enter 1 if OpenSearch _id often equals PK|SK.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Idempotent upsert+version — 1

Why: Safe.

What to do: Enter 1 if indexing uses idempotent upsert with version.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Alarm IteratorAge — 1

Why: Lag.

What to do: Enter 1 if IteratorAge alarms catch consumer lag.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium DLQ for poison — 1

Why: Failure.

What to do: Enter 1 if poison records go to a DLQ.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Backfill Export then tail — 1

Why: History.

What to do: Enter 1 if backfill uses Export→S3 bulk then stream tail.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Ops YAML

Why: Alarms.

What to do: YAML: lag: IteratorAge, poison: DLQ.

How:
  1. Start from the top-level keys the task names.
  2. Indent child keys with 2 spaces (no tabs).
  3. Fill in each property with the value from the task.
  4. Click Check answer. Key order does not matter.

Hard Allow-list indexed fields — 1

Why: Safety.

What to do: Enter 1 if you allow-list fields and exclude PII/secrets.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard No exact-once myth — 1

Why: Truth.

What to do: Enter 1 if practical design is at-least-once + idempotent consumer.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Delete lab same day — 1

Why: Hygiene.

What to do: Enter 1 if lab-ddq-catalog should be deleted same day.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Link Streams chapter — 1

Why: Reuse.

What to do: Enter 1 if this project links the Streams chapter.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

See all practice tasks in the Task Hub →