Ravindra BagaleCourses & study guides मराठी Track your progress

DynamoDB Monitoring: Troubleshooting Throttling and CloudWatch Metrics

By , trainer LinkedIn Last updated:

Throttling is a symptom, not the disease. Lagos fintech on-call Tunde Okonkwo and Bengaluru SRE Nisha Rao said: We raised WCU twice — still throttled one PK. CloudWatch ConsumedWriteCapacity + ThrottledRequests + SuccessfulRequestLatency pointed to a hot partition, not a small table. Pair with hot partitions and capacity modes.

What you'll learn in this chapter

  • Name the CloudWatch metrics that matter for DynamoDB
  • Tell capacity throttle from hot-partition throttle
  • Build alarms that wake humans for the right reason
  • Run a diagnose → fix → verify loop
  • Complete a monitoring lab and cleanup

1. Everyday example: highway toll gates

The highway has many lanes (partitions). One lane jammed with trucks (one hot PK) while others are empty looks like "not enough road" if you only count total cars (table WCU). CloudWatch is the traffic camera. ThrottledRequests is the red light count. Ravi Deshpande (Pune logistics) fixed a bursty TrackShipment PK with write sharding after the cameras showed the truth.

2. Vertical steps

Why -> What -> How -> Lab -> CleanupWhyWhatHowLabCleanup
Always cleanup lab resources

3. Why do we need this?

Why do we need this?

Users feel 400 throttling errors as "app down". Blindly buying more capacity wastes money when one key is hot. Without metrics and alarms you learn from angry tickets. Monitoring turns DynamoDB from a black box into a tunable system.

4. What is it?

What is it?

CloudWatch namespace AWS/DynamoDB: ConsumedReadCapacityUnits, ConsumedWriteCapacityUnits, Provisioned*, ThrottledRequests, ReadThrottleEvents, WriteThrottleEvents, SuccessfulRequestLatency, SystemErrors, UserErrors, ReturnedItemCount for Query/Scan. Contributor Insights (optional) highlights hottest keys. Alarms + dashboards + runbooks close the loop.

5. How to do it (step by step)

How to do it (step by step)

  1. Put a dashboard: consumed vs provisioned (or monitored on-demand peaks), throttles, latency p99.
  2. Alarm ThrottledRequests > 0 for sustained minutes on production tables.
  3. Alarm SystemErrors and high UserErrors separately.
  4. On throttle: check if consumed ≈ provisioned (scale) OR one partition hot (shard/redesign).
  5. Use CloudWatch Contributor Insights or emit PK hash logs carefully for hot-key proof.
  6. After fix: verify throttle = 0 and latency back to baseline.
  7. Review on-demand vs provisioned monthly using consumed history.

6. Throttle decision tree

SignalLikely causeFirst fix
Consumed ≈ Provisioned, many keysTable capacity shortRaise WCU/RCU or on-demand; autoscaling
Throttle + low overall consumedHot partition / hot keyWrite shard, cache, or redesign PK
High latency, few throttlesSlow Query/Scan or GSI fan-outTighten KeyCondition; avoid Scan
UserErrors spikeValidation, ConditionalCheckFailedFix app conditions / payloads

7. Commands with output and meaning

aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB   --metric-name ThrottledRequests   --dimensions Name=TableName,Value=lab-dds-mon   --start-time 2026-11-01T00:00:00Z --end-time 2026-11-02T00:00:00Z   --period 300 --statistics Sum --region ap-south-1
# Output: Datapoints Sum>0 → throttling windows exist
aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB   --metric-name ConsumedWriteCapacityUnits   --dimensions Name=TableName,Value=lab-dds-mon   --start-time 2026-11-01T00:00:00Z --end-time 2026-11-02T00:00:00Z   --period 60 --statistics Sum
# Meaning: compare shape to throttles — flat low consume + throttles ⇒ hot key
aws cloudwatch put-metric-alarm --alarm-name lab-dds-throttle   --namespace AWS/DynamoDB --metric-name ThrottledRequests   --dimensions Name=TableName,Value=lab-dds-mon   --statistic Sum --period 60 --evaluation-periods 3 --threshold 1   --comparison-operator GreaterThanOrEqualToThreshold   --treat-missing-data notBreaching
# Output: Alarm ARN — wakes you when throttles sustain

get-metric-statistics shows history. Compare consumed vs throttle shape. put-metric-alarm makes the pager honest. Delete lab alarms after practice.

8. Industry stories

  • Fintech (Tunde Okonkwo): ledger table throttled one accountId; sharded writes fixed it.
  • SRE (Nisha Rao): on-demand still throttled during extreme burst — smoothed client retries + jitter.
  • Retail (Aisha Khan): flash sale GSI throttled; projected fewer attrs + higher GSI capacity.
  • IoT (Mei Lin): device#day hot at midnight rollover; staggered TTL and shard.
  • Media (Priya Nair): Scan job caused UserErrors and latency — moved to export.
  • Gaming (Carlos Rivera): leaderboard sparse GSI + throttle alarms before launch.

9. Lab

How to do it (step by step)

  1. Create lab-dds-mon on-demand; put a burst of writes to one PK.
  2. Pull ThrottledRequests and ConsumedWriteCapacityUnits for the last hour.
  3. Create a throttle alarm (then delete it).
  4. Delete the table.

Cleanup (stop billing)

  1. Delete lab-dds-mon.
  2. Delete lab-dds-throttle alarm.
  3. Confirm resources gone.

10. Ops checklist before you ship

  1. Confirm TTL attribute name and units (epoch seconds).
  2. Decide soft-delete vs hard-delete vs archive-to-S3.
  3. Enable PITR on any table you cannot rebuild from source.
  4. Document RPO/RTO and who runs restore drills.
  5. If multi-region: pick write stickiness and conflict rules.
  6. If DAX: prove cache-miss path and warm-up plan.
  7. Tag Owner + Expiry on every lab resource.
  8. Budget alert in the account before load tests.

11. Copy-paste DynamoDB CLI starters (batch 2)

export AWS_DEFAULT_REGION=ap-south-1

# TTL status
aws dynamodb describe-time-to-live --table-name ShopApp

# PITR / continuous backups
aws dynamodb describe-continuous-backups --table-name ShopApp

# On-demand backup
aws dynamodb create-backup --table-name ShopApp --backup-name lab-shop-$(date +%Y%m%d)

# List backups
aws dynamodb list-backups --table-name ShopApp

# Global table description (v2)
aws dynamodb describe-table --table-name ShopApp --query 'Table.Replicas'

12. More industry stories

  • Retail (Aisha Khan, Dubai): TTL clears abandoned carts; soft-delete keeps 30-day undo.
  • Bank (Lucas Meyer, Berlin): PITR on ledger table; restore drill every quarter.
  • Travel (Omar Haddad, Amman): Global Tables for catalog reads; inventory writes sticky.
  • Gaming (Carlos Rivera, Mexico City): DAX for hot profile Gets; never cache money ledgers.
  • Social (Priya Nair, Kochi): adjacency edges for follows; sparse GSI for unread inbox.
  • IoT (Mei Lin, Shenzhen): device#day PK + TTL 14 days; rollups to S3.

13. Glossary (batch 2)

TermMeaning
TTLTime to Live — attribute with epoch seconds; DynamoDB deletes later
Soft deleteFlag deleted=true; item remains until TTL/archive
PITRPoint-in-time recovery — continuous backups window
Global TableMulti-region replicated DynamoDB table
LWWLast-writer-wins conflict resolution on replicas
DAXDynamoDB Accelerator — in-memory cache cluster
Adjacency listEdge items modeling relationships in one table
Fan-outWrite once; push copies to many feeds/inboxes

14. Practice: explain out loud

  1. 60s: TTL vs soft-delete vs S3 archive.
  2. 60s: PITR vs on-demand backup.
  3. 60s: when Global Tables help and hurt.
  4. 60s: DAX consistency trade-off.
  5. 60s: CartLite checkout keys in one breath.
  6. 60s: follow edge vs feed item.
  7. Record once; cut filler; record again.

15. Safe lab hygiene

  • Use ap-south-1 labs with a budget alert.
  • Prefix lab tables with lab- and your initials.
  • Delete replicas, DAX clusters, and backups you created for labs.
  • Never enable Global Tables on production without a write plan.
  • Tag every resource Owner + Expiry.
  • Export/import labs can create large S3 bills — delete prefixes.

17. Delete / archive decision pad

NeedPreferWhy
Undo 7-30 daysSoft-delete flagUser restore
Hard expiry, no readTTL attributeAuto delete later
Compliance keep coldExport/archive to S3Cheap long store
Mistake recoveryPITR / backupTable-level restore

18. Ten oral drill questions

  1. What units does DynamoDB TTL use?
  2. Does TTL delete instantly?
  3. PITR window typical length?
  4. Global Tables conflict default?
  5. When sticky writes matter?
  6. DAX vs API Gateway cache?
  7. Cart PK for a shopper?
  8. Order lines under which PK?
  9. Follow edge SK example?
  10. Unread inbox sparse GSI idea?

19. RPO / RTO mini workshop

  1. Write the worst data loss you can accept (RPO) in minutes.
  2. Write how long restore may take (RTO) before business pain.
  3. Map: on-demand backup vs PITR vs multi-region replica.
  4. Schedule a restore drill on a lab table this month.
  5. Name the person who runs the drill.

20. Cost watch list (batch 2)

  • PITR storage and on-demand backup storage.
  • Global Tables: replicated WCUs in every region.
  • DAX node hours even at idle.
  • Export to S3 + S3 storage + Athena later.
  • Streams + Lambda still bill during TTL deletes.

21. Project Definition of Done

  1. Access patterns written and peer-reviewed.
  2. Sample items drawn for happy path + one failure.
  3. Lab table created, exercised, deleted (or tagged Expiry).
  4. At least one conditional write or transaction exercised.
  5. Cleanup commands pasted in your notes.
  6. One interview answer recorded out loud.
  • TTL hygiene → Backup/PITR for “oops” recovery.
  • Global Tables → sticky writes from capacity/hot lessons.
  • DAX → only after Query shapes are correct.
  • P1 CartLite reuses modeling + transactions.
  • P2 Social reuses adjacency + Streams ideas.

23. Quick self-check (yes/no)

  1. I can explain TTL delay vs instant DeleteItem.
  2. I know when soft-delete beats hard delete.
  3. I can start PITR and say what restore costs in time.
  4. I can name one reason NOT to use Global Tables.
  5. I know DAX does not fix a bad partition key.
  6. I can sketch CartLite USER# and ORD# keys.
  7. I can sketch a follow edge item.
  8. I delete lab replicas/DAX/backups the same day.

24. Common errors and fixes

Each row is a real mistake students and juniors hit. Read the fix before you panic.

Error / symptomFix
Only watch table WCUWatch throttles + latency too.
Scale on every spikeCheck hot key first.
Alarm on 1 datapoint noiseUse evaluation periods.
Ignore GSI metricsGSIs throttle separately.
No runbook on alarmLink diagnose tree.
Scan to "debug" prodQuery + metrics instead.
Missing dimension TableNameAlways dimension.
Treat ConditionalCheckFailed as throttleThat is UserErrors.
No post-fix verifyConfirm throttle flat.

25. Troubleshooting checklist

  1. Throttle + low consume: hot partition — shard or redesign.
  2. Throttle + high consume: raise capacity or on-demand.
  3. Empty metrics: wrong region/name/dimension.
  4. Alarm flapping: lengthen period / datapoints.
  5. Latency high without throttle: heavy Query or network.

26. FAQ

Q. Should I always use a single table?

Often yes for related access patterns; not a religion. Separate tables are fine for unrelated domains or different capacity/security needs.

Q. Do I still need RDS?

Yes for heavy joins, multi-row SQL reports, and strong relational constraints. See the intro DynamoDB vs RDS chapter.

Q. Is Scan evil?

Scan is a tool for rare admin jobs and small tables. Design so hot paths use Query/GetItem.

Q. Where are Put/Get command details?

In the CRUD survey, CLI and SDK chapters — this deep set links them instead of copying.

Q. Does on-demand never throttle?

It scales automatically but extreme instant spikes and hot partitions can still throttle.

Q. Contributor Insights cost?

Optional paid feature — use when you need hottest keys; start with metrics+logs.

Q. Which latency metric?

SuccessfulRequestLatency with GetItem/Query operations; watch p99.

Q. Alarm to Slack/Pager?

SNS topic on alarm → chat/pager integration.

Q. Name three must-watch DynamoDB metrics.

Consumed R/W, ThrottledRequests, SuccessfulRequestLatency (plus errors).

Q. How do you spot a hot partition from metrics?

Throttles while table-wide consumed is well under provisioned/peak.

Q. First fix for capacity shortfall vs hot key?

Scale/on-demand vs shard/redesign.

Q. MNC interview: debug production throttling.

Dashboard → throttle vs consumed → GSI → hot key proof → fix → verify alarm clear.

Try at home

  1. Draw a 4-panel dashboard for your table.
  2. Write a 5-line throttle runbook.
  3. List which GSIs need their own alarms.

In short

  1. Watch consumed, throttles, latency, errors.
  2. Throttle + low consume ⇒ hot key.
  3. Throttle + high consume ⇒ capacity.
  4. Alarms need runbooks and verify step.
  5. Link back to keys and capacity chapters.

Revision, interview and comparison cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

Revision card: Throttle metrics. Users see 400s before ops notices capacity or hot keys. Watch ThrottledRequests, ConsumedRead/WriteCapacityUnits, SuccessfulRequestLatency. Alarm on throttle → check hot partition → raise WCU or redesign keys. Ops runbooks open CloudWatch before guessing the SDK.
Revision card: Throttle metrics · Download PNG · Open card page
Revision card: Hot partition. One PK absorbs all traffic and throttles the whole app. Shard write keys or use write sharding; verify with Contributor Insights. Enable Insights → find top keys → add shard suffix → retest RPS. Interview: throttle ≠ always “raise provisioned” — check the key.
Revision card: Hot partition · Download PNG · Open card page
Comparison: Raise WCU only vs Shard hot key. Raise WCU only: ProvisionedThroughputExceeded → bump WCU forever. Shard hot key: Contributor Insights → shard write key then capacity.
Comparison: Raise WCU only vs Shard hot key · Download PNG · Open card page
Comparison: Average consumed only vs Contributor Insights. Average consumed only: Dashboard shows average ConsumedWrite under limit. Contributor Insights: Insights lists top keys causing throttle.
Comparison: Average consumed only vs Contributor Insights · Download PNG · Open card page
Interview card: Must-watch metrics. Only CPU-style myths miss DynamoDB pain signals. ThrottledRequests, Consumed*Capacity, SystemErrors, UserErrors, Latency. MNC: name three metrics you graph on day-1 go-live.
Interview card: Must-watch metrics · Download PNG · Open card page
Interview card: Useful alarms. Alarms on 100% CPU copy-paste never fire for DDB. Alarm ThrottledRequests >0 sustained; page on SystemErrors spike. MNC: show SNS → Slack path for throttle alarm.
Interview card: Useful alarms · Download PNG · Open card page
Interview card: Throttle fix. Raising RCU forever hides a hot partition. Confirm throttle metric → Insights top keys → shard or GSI → then capacity. MNC: walk a ProvisionedThroughputExceededException root-cause.
Interview card: Throttle fix · Download PNG · Open card page
Interview card: Contributor Insights. Average metrics hide the one hot PK. Enable Contributor Insights; act on most-accessed keys. MNC: enable Insights, name the top write key, propose shard.
Interview card: Contributor Insights · Download PNG · Open card page

More cards: Ravi Sir's Cards.

Practice Tasks

About this scenario

Vidarbha Cotton Mills is a textiles business in Nagpur. Ganesh Wankhede looks after its IT and cloud work. They count money in INR and run their AWS resources in the ap-south-1 Region. In the tasks below you do real jobs for them using what this lesson teaches. All names, account IDs, IP addresses and numbers are made up for practice.

0 / 16

Easy Watch ThrottledRequests — 1

Why: Metric.

What to do: Enter 1 if dashboards must include ThrottledRequests.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy Low consume + throttle = hot — 1

Why: Signal.

What to do: Enter 1 if throttle with low consumed often means hot key.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy High consume + throttle = scale — 1

Why: Signal.

What to do: Enter 1 if throttle near provisioned means raise capacity/on-demand.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy Watch latency p99 — 1

Why: Metric.

What to do: Enter 1 if SuccessfulRequestLatency p99 matters.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Easy Describe mon table

Why: CLI.

What to do: Describe table ShopApp

How:
  1. Read the task and decide which AWS service it is about.
  2. Write aws, then the service name, then the operation (the action word).
  3. Add each needed flag (--name) followed by its value from the task.
  4. Flags can be in any order. Click Check answer.

Easy Metrics YAML

Why: Names.

What to do: YAML: throttle: ThrottledRequests, consumed: ConsumedWriteCapacityUnits.

How:
  1. Start from the top-level keys the task names.
  2. Indent child keys with 2 spaces (no tabs).
  3. Fill in each property with the value from the task.
  4. Click Check answer. Key order does not matter.

Easy No prod Scan debug — 1

Why: Anti.

What to do: Enter 1 if you should not Scan prod to debug throttles.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium GSI throttles separately — 1

Why: Scope.

What to do: Enter 1 if GSIs can throttle on their own.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium On-demand can still throttle — 1

Why: Truth.

What to do: Enter 1 if on-demand can still throttle on extreme bursts/hot keys.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Eval periods on alarms — 1

Why: Hygiene.

What to do: Enter 1 if throttle alarms need evaluation periods.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium CCF is UserErrors — 1

Why: Classify.

What to do: Enter 1 if ConditionalCheckFailed is UserErrors not throttle.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Medium Fix YAML

Why: Tree.

What to do: YAML: hot: shard, short: scale.

How:
  1. Start from the top-level keys the task names.
  2. Indent child keys with 2 spaces (no tabs).
  3. Fill in each property with the value from the task.
  4. Click Check answer. Key order does not matter.

Hard Alarm needs runbook — 1

Why: Ops.

What to do: Enter 1 if every prod throttle alarm links a runbook.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Verify after fix — 1

Why: Close.

What to do: Enter 1 if you re-check metrics until throttles clear.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Dimension TableName — 1

Why: CW.

What to do: Enter 1 if metrics need TableName dimension.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

Hard Delete lab same day — 1

Why: Hygiene.

What to do: Enter 1 if lab-dds-mon should be deleted same day.

How:
  1. Write down the numbers the task gives you.
  2. Work it out step by step on paper.
  3. Type only the final answer in the box.
  4. Click Check answer.

See all practice tasks in the Task Hub →