Queue-Based Load Leveling, DLQ, Retries and Idempotency on Serverless

By Ravindra Bagale, trainer · LinkedIn · Last updated: 2 November 2026

1. Everyday example: ticket rail

Sofia's cafe in Sofia (EUR) takes 40 orders when a tour bus arrives. The barista cannot froth 40 milks in 40 seconds. Tickets hang on a rail; they cook at a human pace. SQS is that rail for Lambda: API returns 202, workers pull when ready.

Build it hands-on in async order project. Messaging depth in SQS/SNS.

2. Why load leveling?

Why do we need this?

Without a queue, every spike becomes a concurrency spike, then throttles, then angry clients retrying harder. A queue turns a spike into lag — measurable, alarmable, drainable.

What is it?

Queue-based load leveling: producers enqueue work; consumers process at a controlled rate. DLQ holds messages that failed too many times. Idempotency makes at-least-once delivery safe.

3. Vertical path

4. Visibility timeout vs Lambda timeout

5. How: wire intake + worker

How to do it (step by step)

1. Create main queue + DLQ; set redrive maxReceiveCount.
2. Intake Lambda: validate → SendMessage → 202 + id.
3. Worker Lambda: parse → idempotent PutItem → success deletes message (ESM does it).
4. Event source mapping with partial batch failure reporting on.
5. Alarm: DLQ depth > 0; oldest message age high.
6. Load test burst; watch concurrency and lag.

Create queue with redrive (shape)

aws sqs create-queue --queue-name sofia-orders-dlq --region ap-south-1
# then create main with RedrivePolicy pointing at DLQ ARN, maxReceiveCount=5

Meaning: Always create DLQ first so the ARN exists.

Send a message

aws sqs send-message --queue-url $Q --region ap-south-1 \
  --message-body '{"orderId":"o-7","currency":"EUR","total":12.5}'

Meaning: Worker should create DynamoDB row once even if you send twice with same orderId (idempotent).

6. Idempotency patterns

Python: conditional put

# Concept: PutItem with ConditionExpression attribute_not_exists(orderId)
# First write wins; duplicate message returns ConditionalCheckFailed — treat as success
print("first -> wrote")
print("dup -> already exists -> ok")

Meaning: at-least-once delivery becomes exactly-once effect on your table.

Node: hash payload key

const crypto = require('crypto');
const key = (o) => crypto.createHash('sha256').update(JSON.stringify(o)).digest('hex').slice(0,16);
console.log(key({orderId:'o-7',total:12.5}));

Meaning: stable keys help dedupe when producers forget to send the same id.

7. Retry taxonomy

8. Partial batch failure

When batch size > 1, one poison item should not rewind the whole batch forever. Report failures item-by-item so successes delete and poison heads to DLQ.

9. Lab: paper timeouts + optional queue pair

Do this in ap-south-1 (Mumbai) unless your org forbids it. Set a budget alert first.

How to do it (step by step)

1. Write Lambda timeout and visibility timeout numbers for a 2s p99 worker.
2. Draw maxReceiveCount path to DLQ.
3. Optional sandbox: queue+DLQ+send+receive once.
4. Alarm sketch: DLQ > 0 for 1 minute.
5. Cleanup queues.

Cleanup (stop billing)

1. Delete main queue then DLQ.
2. Delete lab function and log group.

10. FIFO vs Standard for leveling

• Standard: higher throughput, best-effort ordering, duplicates possible.
• FIFO: order per message group, exactly-once processing within constraints, lower throughput.
• Aisha in Lagos uses FIFO for ledger postings (NGN); Standard for email notifications.

Get redrive + visibility

aws sqs get-queue-attributes --queue-url $Q --attribute-names All --region ap-south-1 --query 'Attributes.{V:VisibilityTimeout,R:RedrivePolicy,D:DelaySeconds}'

Meaning: Confirm visibility and redrive before load test.

11. Backpressure to producers

If lag grows without bound, stop accepting or shed load at API throttle. A queue is a shock absorber, not infinite disk for a broken worker.

12. Same pattern across industries

• Fintech (Aisha, Lagos): payment webhooks → SQS → Lambda; never block the bank callback thread.
• Ecommerce (Rohan, Pune): checkout API on API Gateway+Lambda; inventory fan-out on SNS.
• Healthcare (Mei, Singapore): appointment SMS via SNS; PHI stays in DynamoDB with fine-grained IAM.
• EdTech (Diego, São Paulo): nightly cleanup EventBridge → Lambda on student drafts.
• Logistics (Fatima, Casablanca): package scan events on EventBridge bus; Step Functions for customs saga.
• Media (Kenji, Osaka): S3 upload → Lambda thumbnails; fan-out to moderation + CDN invalidate.
• SaaS multi-tenant (Priya, Bengaluru): Cognito groups + DynamoDB partition per tenant.
• Gov portal (Omar, Amman): strangler fig — peel one PDF form route per sprint to Lambda.

13. Troubleshooting checklist

1. Confirm region (ap-south-1 vs us-east-1 for CloudFront/ACM quirks).
2. Check CloudWatch Logs for the function name and request id.
3. Verify IAM execution role allows the API you call.
4. For async: check DLQ depth and SQS age of oldest message.
5. For APIs: curl -v and compare status, CORS headers, body.
6. For workflows: open Step Functions execution event history.
7. Re-read the linked deep chapter before reinventing a service.
8. If cost spikes: check concurrency, retries, and provisioned capacity.

14. Common errors and fixes

Each row is a real mistake students and juniors hit. Read the fix before you panic.

15. FAQ

16. Interview Q&A (Beginner / Mid / Senior + MNC)

17. Tips in English, Marathi and Hindi

18. Architecture decision checklist

1. Write the user journey in five bullets with no AWS names.
2. Mark each step sync (user waits) or async.
3. Name failure mode: retry, DLQ, compensate, or alert-only.
4. Name the idempotency key for every write.
5. Pick data store from access patterns (link DynamoDB modeling).
6. Estimate monthly cost at 1x and 10x traffic (USD and INR).
7. Confirm account concurrency and service quotas in ap-south-1.
8. Decide auth: Cognito, IAM, Lambda authorizer, or none.
9. Name on-call and which CloudWatch alarm pages them.
10. Link the deep chapter you will not rewrite.
11. Only then open console or IaC.

19. Copy-paste quick commands (read-only first)

# Region and identity
aws configure get region
aws sts get-caller-identity

# Lambda landscape
aws lambda list-functions --region ap-south-1 --max-items 20 \
  --query 'Functions[].{N:FunctionName,R:Runtime,T:Timeout,M:MemorySize}' --output table
aws lambda get-account-settings --region ap-south-1

# API Gateway (HTTP APIs)
aws apigatewayv2 get-apis --region ap-south-1 --query 'Items[].{N:Name,Id:ApiId,Ep:ProtocolType}' --output table

# Queues and topics
aws sqs list-queues --region ap-south-1
aws sns list-topics --region ap-south-1 --query 'Topics[].TopicArn'

# Event buses and state machines
aws events list-event-buses --region ap-south-1
aws stepfunctions list-state-machines --region ap-south-1 --query 'stateMachines[].name'

# DynamoDB tables (names only)
aws dynamodb list-tables --region ap-south-1

Run these in a sandbox account. Prefer read-only IAM while learning.

20. More industry and stack stories

• Fintech (Aisha, Lagos, NGN): bank webhooks must return 200 fast — SQS absorbs spikes; ledger Lambda is idempotent on paymentId.
• Ecommerce (Rohan, Pune, INR): Black Friday — API Gateway throttle + SQS order intake; SNS fans inventory, email, analytics.
• Healthcare (Mei, Singapore, SGD): appointment reminders via SNS SMS; PHI only in DynamoDB with condition keys.
• EdTech (Diego, São Paulo, BRL): EventBridge nightly cleanup of draft quizzes; Lambda timeout 30s, not 15 min.
• Logistics (Fatima, Casablanca, MAD): package scans on custom EventBridge bus; Step Functions customs saga.
• Media (Kenji, Osaka, JPY): S3 → Lambda thumbnails; never write thumbs to the watched prefix.
• SaaS (Priya, Bengaluru, INR): Cognito groups + DynamoDB PK tenantId; one Lambda authorizer for all routes.
• Gov (Omar, Amman, JOD): strangler — peel one PDF form route per sprint from the monolith.
• Startup (Sofia, Sofia, EUR): start with Function URL for demos; move to HTTP API when partners need JWT.
• Retail (Luis, Mexico City, MXN): store POS sync via SQS FIFO when order matters; Standard when it does not.

• Python: handlers thin; put domain logic in plain modules for unit tests without Lambda.
• Node: prefer async/await; never block the event loop on sync DNS.
• Java: SnapStart for cold starts; keep dependencies lean.
• .NET: AspNetCoreServer for low-traffic tenants; watch package size.
• Go: single static binary; great for SQS workers.
• PHP: keep Laravel queues on ECS; Lambda for S3/webhooks glue.

21. Glossary (short)

22. Practice: explain out loud

1. 60 seconds: everyday story for queue load leveling.
2. 60 seconds: sync vs async with a checkout example.
3. 60 seconds: when you refuse Lambda.
4. 60 seconds: DLQ + idempotency in one breath.
5. Record once; cut fillers; record again.

23. Safe lab hygiene

• Tag every lab resource with owner and expiry date.
• Set a monthly AWS Budget alarm in INR or USD before experiments.
• Prefer read-only commands first; mutate in a sandbox account.
• Delete queues, APIs, tables you created — DynamoDB and APIGW leave footprints.
• Never put production secrets in env vars or screenshots.
• Do not enable provisioned concurrency for a hello-world.

24. How this chapter links others

• Compute depth: Lambda.
• HTTP edge: API Gateway.
• Data modeling: DynamoDB.
• Messaging: SQS/SNS.
• Events: EventBridge.
• Workflows: Step Functions.
• Project chapters (build-*): worked assemblies — link, do not paste.

25. Myths to drop

26. Error budget thinking for serverless

If your monthly error budget is 0.1% on checkout, a fan-out that retries aggressively can burn the budget with duplicate emails while money stays correct. Separate business correctness SLOs from notification SLOs.

1. Write two SLOs: money-correct and notify-delivered.
2. Retries on money path: bounded + idempotent.
3. Retries on notify path: best-effort + DLQ.
4. Page humans on money DLQ; ticket on notify DLQ.

27. Regional notes (ap-south-1)

• Build labs in Mumbai (ap-south-1) unless your org mandates elsewhere.
• Some services launch first in us-east-1 — check feature availability.
• Data residency for INR fintech may require in-region only.
• Latency to end users in India usually favors ap-south-1.

28. Naming conventions that save incidents

# Suggested prefixes
# Lambda:  pay-intake-prod, pay-worker-prod
# SQS:     pay-orders.fifo / pay-orders-dlq
# SNS:     pay-order-events
# EB bus:  pay.domain
# SFN:     pay-refund-saga
# DDB:     pay-orders-prod

Include env (dev/stage/prod) and domain. Future you grepping CloudWatch will thank present you.

29. Pre-mortems (ten minutes)

1. Imagine tomorrow headline: double charge in AED. What broke?
2. Imagine DLQ depth 50,000 after a bad deploy. How do you drain?
3. Imagine Cognito outage. Do APIs fail closed or serve read-only?
4. Write answers in the runbook before go-live.

30. Currency and locale in examples

Rotate currencies in designs: INR, USD, EUR, AED, NGN, BRL, JPY, SGD. Rotate names: Aisha, Rohan, Mei, Diego, Fatima, Kenji, Priya, Omar, Sofia, Luis. This keeps mental models global — not one-country samples.

31. Worked failure timeline

1. Message arrives; worker throws on bad JSON.
2. Visibility expires or failure reported; receive count +1.
3. After maxReceiveCount, message to DLQ.
4. Alarm fires; human inspects body.
5. Fix code; redrive; confirm DynamoDB.

32. Metrics that matter

• ApproximateNumberOfMessagesVisible — backlog.
• ApproximateAgeOfOldestMessage — SLA risk.
• NumberOfMessagesSent vs Deleted — leak detection.
• Lambda ConcurrentExecutions + Throttles — consumer side.

Revision, interview and comparison cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

More cards: Ravi Sir's Cards.