Ravindra BagaleCourses & study guides मराठी Track your progress

Project: IoT Time-Series, TTL and Rollups on DynamoDB

By , trainer LinkedIn Last updated:

IoT never stops writing. Bengaluru ops lead Ananya Iyer and fleet PM Diego Ruiz ship PulseLite: sharded DEVICE# keys, TTL on raw samples, hourly ROLLUP# items. Reuse TTL and Streams; link sharding.

What you'll learn in this chapter

  • Shard device write keys
  • TTL raw telemetry
  • Build hourly rollups
  • Industry parallels
  • Lab + cleanup

1. Everyday example: weather station notebook

Every minute a new line; weekly you keep daily averages and toss raw pages. Raw = TTL. Averages = rollups. Many stations = shard the notebook spines.

2. Vertical steps

Patterns -> Shard keys -> Ingest+TTL -> Rollup worker -> CleanupPatternsShard keysIngest+TTLRollup workerCleanup
Cleanup

3. Why do we need this?

Why do we need this?

One DeviceId PK under burst throttles; eternal raw points burn storage. Interviews ask hot partitions + TTL + aggregate design together.

4. What is it?

What is it?

PulseLite: PK=DEVICE#id#shard SK=epochMs (metrics, expires_at TTL); PK=DEVICE#id SK=ROLLUP#yyyyMMddHH (count/sum/min/max). Streams Lambda upserts rollups idempotently.

5. How to do it (step by step)

How to do it (step by step)

  1. List patterns: ingest, last 15m, hourly chart, device meta.
  2. Pick shard count (start 4); SK=epoch ms.
  3. Create lab-ddn-telemetry; enable TTL on expires_at.
  4. Put 200 points across shards with short TTL.
  5. Paper/Lambda: aggregate to ROLLUP#hour (idempotent).
  6. Query rollup; confirm raw shrinks; delete table.

6. Key sketch

EntityPKSK
Raw pointDEVICE#d1#21730000000123
Hour rollupDEVICE#d1ROLLUP#2026110214
Device metaDEVICE#d1META

7. Commands with output and meaning

aws dynamodb create-table --table-name lab-ddn-telemetry \
  --attribute-definitions AttributeName=PK,AttributeType=S AttributeName=SK,AttributeType=S \
  --key-schema AttributeName=PK,KeyType=HASH AttributeName=SK,KeyType=RANGE \
  --billing-mode PAY_PER_REQUEST --region ap-south-1
aws dynamodb update-time-to-live --table-name lab-ddn-telemetry \
  --time-to-live-specification "Enabled=true, AttributeName=expires_at"
aws dynamodb put-item --table-name lab-ddn-telemetry --item '{
 "PK":{"S":"DEVICE#d1#0"},"SK":{"S":"1730000000123"},
 "temp":{"N":"31.2"},"expires_at":{"N":"1730003600"}}'
aws dynamodb query --table-name lab-ddn-telemetry \
  --key-condition-expression "PK = :pk AND SK BETWEEN :a AND :b" \
  --expression-attribute-values '{":pk":{"S":"DEVICE#d1#0"},":a":{"S":"1730000000000"},":b":{"S":"1730001000000"}}'
# Output: window of points. Meaning: time Query on shard; rollups queried separately.

TTL seconds not ms; shard before load test; rollup via Streams (link) not in Put path.

8. Industry stories

  • Manufacturing (Ananya): vibration samples + hourly RMS.
  • Fleet (Diego): GPS pings; daily distance rollup.
  • Energy (Omar): meter 1-min; billing daily.
  • Agri (Sofia): soil moisture alerts.
  • Health (Meera): wearable HR daily summary.
  • Smart city (Priya): parking occupancy rollups.

9. Lab

How to do it (step by step)

  1. Create lab-ddn-telemetry; enable TTL; put sharded points.
  2. Write rollup items by hand or tiny script; Query charts.
  3. Delete table same day.

Cleanup (stop billing)

  1. Delete lab resources you created.
  2. Confirm with describe that names are gone.

10. Ops checklist before you ship

  1. Confirm TTL attribute name and units (epoch seconds).
  2. Decide soft-delete vs hard-delete vs archive-to-S3.
  3. Enable PITR on any table you cannot rebuild from source.
  4. Document RPO/RTO and who runs restore drills.
  5. If multi-region: pick write stickiness and conflict rules.
  6. If DAX: prove cache-miss path and warm-up plan.
  7. Tag Owner + Expiry on every lab resource.
  8. Budget alert in the account before load tests.

11. Copy-paste DynamoDB CLI starters (batch 2)

export AWS_DEFAULT_REGION=ap-south-1

# TTL status
aws dynamodb describe-time-to-live --table-name ShopApp

# PITR / continuous backups
aws dynamodb describe-continuous-backups --table-name ShopApp

# On-demand backup
aws dynamodb create-backup --table-name ShopApp --backup-name lab-shop-$(date +%Y%m%d)

# List backups
aws dynamodb list-backups --table-name ShopApp

# Global table description (v2)
aws dynamodb describe-table --table-name ShopApp --query 'Table.Replicas'

12. More industry stories

  • Retail (Aisha Khan, Dubai): TTL clears abandoned carts; soft-delete keeps 30-day undo.
  • Bank (Lucas Meyer, Berlin): PITR on ledger table; restore drill every quarter.
  • Travel (Omar Haddad, Amman): Global Tables for catalog reads; inventory writes sticky.
  • Gaming (Carlos Rivera, Mexico City): DAX for hot profile Gets; never cache money ledgers.
  • Social (Priya Nair, Kochi): adjacency edges for follows; sparse GSI for unread inbox.
  • IoT (Mei Lin, Shenzhen): device#day PK + TTL 14 days; rollups to S3.

13. Glossary (batch 2)

TermMeaning
TTLTime to Live — attribute with epoch seconds; DynamoDB deletes later
Soft deleteFlag deleted=true; item remains until TTL/archive
PITRPoint-in-time recovery — continuous backups window
Global TableMulti-region replicated DynamoDB table
LWWLast-writer-wins conflict resolution on replicas
DAXDynamoDB Accelerator — in-memory cache cluster
Adjacency listEdge items modeling relationships in one table
Fan-outWrite once; push copies to many feeds/inboxes

14. Practice: explain out loud

  1. 60s: TTL vs soft-delete vs S3 archive.
  2. 60s: PITR vs on-demand backup.
  3. 60s: when Global Tables help and hurt.
  4. 60s: DAX consistency trade-off.
  5. 60s: CartLite checkout keys in one breath.
  6. 60s: follow edge vs feed item.
  7. Record once; cut filler; record again.

15. Safe lab hygiene

  • Use ap-south-1 labs with a budget alert.
  • Prefix lab tables with lab- and your initials.
  • Delete replicas, DAX clusters, and backups you created for labs.
  • Never enable Global Tables on production without a write plan.
  • Tag every resource Owner + Expiry.
  • Export/import labs can create large S3 bills — delete prefixes.

17. Delete / archive decision pad

NeedPreferWhy
Undo 7-30 daysSoft-delete flagUser restore
Hard expiry, no readTTL attributeAuto delete later
Compliance keep coldExport/archive to S3Cheap long store
Mistake recoveryPITR / backupTable-level restore

18. Ten oral drill questions

  1. What units does DynamoDB TTL use?
  2. Does TTL delete instantly?
  3. PITR window typical length?
  4. Global Tables conflict default?
  5. When sticky writes matter?
  6. DAX vs API Gateway cache?
  7. Cart PK for a shopper?
  8. Order lines under which PK?
  9. Follow edge SK example?
  10. Unread inbox sparse GSI idea?

19. RPO / RTO mini workshop

  1. Write the worst data loss you can accept (RPO) in minutes.
  2. Write how long restore may take (RTO) before business pain.
  3. Map: on-demand backup vs PITR vs multi-region replica.
  4. Schedule a restore drill on a lab table this month.
  5. Name the person who runs the drill.

20. Cost watch list (batch 2)

  • PITR storage and on-demand backup storage.
  • Global Tables: replicated WCUs in every region.
  • DAX node hours even at idle.
  • Export to S3 + S3 storage + Athena later.
  • Streams + Lambda still bill during TTL deletes.

21. Project Definition of Done

  1. Access patterns written and peer-reviewed.
  2. Sample items drawn for happy path + one failure.
  3. Lab table created, exercised, deleted (or tagged Expiry).
  4. At least one conditional write or transaction exercised.
  5. Cleanup commands pasted in your notes.
  6. One interview answer recorded out loud.
  • TTL hygiene → Backup/PITR for “oops” recovery.
  • Global Tables → sticky writes from capacity/hot lessons.
  • DAX → only after Query shapes are correct.
  • P1 CartLite reuses modeling + transactions.
  • P2 Social reuses adjacency + Streams ideas.

23. Quick self-check (yes/no)

  1. I can explain TTL delay vs instant DeleteItem.
  2. I know when soft-delete beats hard delete.
  3. I can start PITR and say what restore costs in time.
  4. I can name one reason NOT to use Global Tables.
  5. I know DAX does not fix a bad partition key.
  6. I can sketch CartLite USER# and ORD# keys.
  7. I can sketch a follow edge item.
  8. I delete lab replicas/DAX/backups the same day.

24. Common errors and fixes

Each row is a real mistake students and juniors hit. Read the fix before you panic.

Error / symptomFix
Hot single DeviceId PKAdd #shard suffix.
TTL in millisecondsEpoch seconds Number.
Fat 400KB blobsS3 pointer in item.
Sync rollup in API pathAsync Streams/worker.
Duplicate ingestIdempotent SK or condition.
Scan for dashboardsQuery rollup keys.
Clock-skew expires_atServer sets expiry.
Leave lab tableDelete same day.
Rewrite TTL chapterLink TTL + Streams.

25. Troubleshooting checklist

  1. Throttle on one device: raise shards.
  2. TTL not firing: Number epoch + enable TTL.
  3. Rollup mismatch: idempotent sum/max + replay DLQ.
  4. Empty Query: same epoch units on SK.
  5. Stream lag: IteratorAge + parallelization.

26. FAQ

Q. Should I always use a single table?

Often yes for related access patterns; not a religion. Separate tables are fine for unrelated domains or different capacity/security needs.

Q. Do I still need RDS?

Yes for heavy joins, multi-row SQL reports, and strong relational constraints. See the intro DynamoDB vs RDS chapter.

Q. Is Scan evil?

Scan is a tool for rare admin jobs and small tables. Design so hot paths use Query/GetItem.

Q. Where are Put/Get command details?

In the CRUD survey, CLI and SDK chapters — this deep set links them instead of copying.

Q. Timestream instead?

Often both: DynamoDB ops latest; Timestream/S3 deep history.

Q. How many shards?

Start small; scale with throttle metrics.

Q. GSI on every point?

Prefer rollup keys; sparse if needed.

Q. Export cold?

Export to S3 before aggressive TTL.

27. Interview Q&A

Q. (B) Raw key shape?

DEVICE#id#shard / epochMs.

Q. (B) Why TTL raw?

Cap storage; keep rollups.

Q. (M) Rollup how?

Streams/worker idempotent upsert.

Q. (M) Hot device?

More shards / buffers.

Q. (S/MNC) Cost control?

TTL+rollups+on-demand+S3 archive.

Q. (MNC) Out-of-order?

SK=ts; versioned rollup merge.

28. Tips in English, Marathi and Hindi

Ravindra Bagale's Tip (EN)

Shard writes, TTL raw, Query rollups — never Scan the ocean.

Ravindra Bagale's Tip (MR)

Shard + TTL raw + rollup Query.

Tip (HI)

शार्ड लिखें, raw पर TTL, rollup Query करें।

Try at home

  1. Pick shard count for a 1k msg/s device.
  2. Lab put+TTL+rollup item.
  3. List metrics to alarm.

In short

  1. PulseLite = sharded raw + ROLLUP#.
  2. TTL caps raw storage.
  3. Async idempotent rollups.
  4. Cleanup labs.

Revision, interview and comparison cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

Revision card: Shard writes. One DeviceId PK throttles under burst. PK=DEVICE#id#shard; SK=epochMs; raise shards on throttle. Pick shard count → Put across shards → Query window → watch throttle. Ananya Iyer starts at 4 shards and scales with metrics.
Revision card: Shard writes · Download PNG · Open card page
Revision card: TTL + rollups. Eternal raw points burn storage; Scan dashboards die. TTL raw; async ROLLUP#hour items for charts. Enable TTL → Streams worker → Query ROLLUP# → export cold. Diego Ruiz never Scans raw for hourly charts.
Revision card: TTL + rollups · Download PNG · Open card page
Comparison: Raw only vs Rollup items. Raw only: Dashboards Scan endless raw points. Rollup items: Query ROLLUP#hour aggregate items.
Comparison: Raw only vs Rollup items · Download PNG · Open card page
Comparison: Single device PK vs Sharded PK. Single device PK: DEVICE#id alone heats under burst. Sharded PK: DEVICE#id#shard spreads writes.
Comparison: Single device PK vs Sharded PK · Download PNG · Open card page
Interview card: Enable TTL. Attribute without enable never expires. update-time-to-live on expires_at Number seconds. Create lab → enable → put short TTL → describe → delete.
Interview card: Enable TTL · Download PNG · Open card page
Interview card: Pick shards. Guessing shard=1 under burst fails. Start small; raise shards when ThrottledRequests rise. Ingest test → watch metrics → bump shard → document.
Interview card: Pick shards · Download PNG · Open card page
Interview card: Hourly rollup. Sync aggregate in API path slows ingest. Streams/worker upserts ROLLUP#yyyyMMddHH idempotently. Paper worker → Put rollup → Query chart → replay DLQ story.
Interview card: Hourly rollup · Download PNG · Open card page
Interview card: Cold export. Aggressive TTL without archive loses history. Export to S3 / lake before short TTL windows. Name retention → export job → TTL policy → verify.
Interview card: Cold export · Download PNG · Open card page

More cards: Ravi Sir's Cards.

Practice Tasks

About this scenario

Vidarbha Cotton Mills is a textiles business in Nagpur. Ganesh Wankhede looks after its IT and cloud work. They count money in INR and run their AWS resources in the ap-south-1 Region. In the tasks below you do real jobs for them using what this lesson teaches. All names, account IDs, IP addresses and numbers are made up for practice.

0 / 16

Easy Shard device PK — 1

Why: Hot.

What to do: Enter 1 if device writes use DEVICE#id#shard keys.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy SK is epoch time — 1

Why: Sort.

What to do: Enter 1 if raw SK is epoch time for window queries.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy TTL raw points — 1

Why: Cost.

What to do: Enter 1 if raw telemetry should TTL.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy Hourly ROLLUP items — 1

Why: Agg.

What to do: Enter 1 if dashboards query ROLLUP#hour items.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy Describe telemetry table

Why: CLI.

What to do: Describe table ShopApp

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Easy Raw key YAML

Why: Sketch.

What to do: YAML: pk: DEVICE#d1#0, sk: 1730000000123.

How:
  1. Start from the top-level keys the task names.
  2. Indent child keys with 2 spaces (no tabs).
  3. Fill in each property with the value from the task.
  4. Click Check answer. Key order does not matter.

Easy Export before hard TTL — 1

Why: Cold.

What to do: Enter 1 if you export cold data before aggressive TTL.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium TTL seconds not ms — 1

Why: Trap.

What to do: Enter 1 if expires_at must be epoch seconds.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Async rollup worker — 1

Why: Path.

What to do: Enter 1 if rollups are async via Streams/worker.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Fat blobs to S3 — 1

Why: Size.

What to do: Enter 1 if huge payloads use S3 pointers.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Do not Scan dashboards — 1

Why: Anti.

What to do: Enter 1 if dashboards should Query rollups not Scan raw.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Rollup YAML

Why: Shape.

What to do: YAML: sk: ROLLUP#2026110214, metric: count.

How:
  1. Start from the top-level keys the task names.
  2. Indent child keys with 2 spaces (no tabs).
  3. Fill in each property with the value from the task.
  4. Click Check answer. Key order does not matter.

Hard Idempotent rollup — 1

Why: CDC.

What to do: Enter 1 if rollup upserts must be idempotent.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard More shards if throttle — 1

Why: Scale.

What to do: Enter 1 if throttles mean raise shard count.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Delete lab same day — 1

Why: Hygiene.

What to do: Enter 1 if lab-ddn-telemetry should be deleted same day.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Link TTL + Streams — 1

Why: Reuse.

What to do: Enter 1 if this project links TTL and Streams chapters.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

See all practice tasks in the Task Hub →