Serverless वर Queue-Based Load Leveling, DLQ, Retries आणि Idempotency

By Ravindra Bagale, trainer · LinkedIn · Last updated: 2 November 2026

1. रोजचे: ticket rail

Sofia (EUR) cafe: tour bus = 40 orders. Ticket rail = SQS. API 202; workers steady.

Hands-on: async order project. Depth: SQS/SNS.

2. Load leveling का?

हे का लागतं?

Queue नसेल तर spike → throttle → angry retries. Queue = lag जो measurable आहे.

हे काय आहे?

Load leveling + DLQ + Idempotency.

3. Vertical path

4. Visibility vs Lambda timeout

5. How: intake + worker

कसं करायचं (step by step)

1. Queue+DLQ redrive.
2. Intake 202.
3. Worker idempotent.
4. Partial batch failure.
5. Alarms.
6. Burst test.

Create + redrive

aws sqs create-queue --queue-name sofia-orders-dlq --region ap-south-1
# then create main with RedrivePolicy pointing at DLQ ARN, maxReceiveCount=5

अर्थ: आधी DLQ ARN.

SendMessage

aws sqs send-message --queue-url $Q --region ap-south-1 \
  --message-body '{"orderId":"o-7","currency":"EUR","total":12.5}'

अर्थ: तेच orderId दोनदा = एक row.

6. Idempotency

Python conditional

# Concept: PutItem with ConditionExpression attribute_not_exists(orderId)
# First write wins; duplicate message returns ConditionalCheckFailed — treat as success
print("first -> wrote")
print("dup -> already exists -> ok")

अर्थ: at-least-once → exactly-once effect.

Node key

const crypto = require('crypto');
const key = (o) => crypto.createHash('sha256').update(JSON.stringify(o)).digest('hex').slice(0,16);
console.log(key({orderId:'o-7',total:12.5}));

अर्थ: stable key = dedupe.

7. Retry taxonomy

8. Partial batch failure

Batch मध्ये एक poison ≠ सगळं rewind. Item-level failure report.

9. Lab

Org block नसेल तर ap-south-1 (Mumbai). आधी budget alert.

कसं करायचं (step by step)

1. Timeout numbers.
2. Redrive path.
3. Optional.
4. Alarm sketch.
5. Cleanup.

Cleanup (billing थांबवा)

1. Main then DLQ.
2. Function+logs.

10. FIFO vs Standard

• Standard: throughput.
• FIFO: order.
• Aisha: FIFO ledger; Standard email.

Attributes

aws sqs get-queue-attributes --queue-url $Q --attribute-names All --region ap-south-1 --query 'Attributes.{V:VisibilityTimeout,R:RedrivePolicy,D:DelaySeconds}'

अर्थ: Load test आधी confirm.

11. Backpressure

Queue = shock absorber; infinite disk नव्हे. API throttle/shed.

12. एकाच pattern — वेगवेगळ्या industries

• Fintech (Aisha, Lagos): payment webhooks → SQS → Lambda.
• Ecommerce (Rohan, Pune): checkout API; inventory SNS fan-out.
• Healthcare (Mei, Singapore): SNS SMS; PHI + fine-grained IAM.
• EdTech (Diego, São Paulo): EventBridge nightly cleanup.
• Logistics (Fatima, Casablanca): EventBridge + saga.
• Media (Kenji, Osaka): S3 → Lambda thumbs + fan-out.
• SaaS (Priya, Bengaluru): Cognito + per-tenant partition.
• Gov (Omar, Amman): strangler fig per form route.

13. Troubleshooting checklist

1. Region confirm करा.
2. CloudWatch Logs + request id.
3. Execution role permissions.
4. DLQ + oldest message age.
5. curl -v + CORS.
6. SFN event history.
7. Linked chapter पुन्हा वाचा.
8. Cost: concurrency/retries.

14. सामान्य errors आणि fixes

प्रत्येक row = खरी चूक. Panic करण्याआधी fix वाचा.

15. FAQ

16. Interview Q&A (Beginner / Mid / Senior + MNC)

17. Tips — English, मराठी आणि हिंदी

18. Architecture decision checklist

1. User journey 5 bullets, no AWS names.
2. Sync vs async mark.
3. Failure mode.
4. Idempotency key.
5. Data from access patterns.
6. Cost 1x and 10x.
7. Quotas ap-south-1.
8. Auth decision.
9. On-call + alarm.
10. Deep chapter link.
11. मग console/IaC.

19. Copy-paste quick commands

# Region and identity
aws configure get region
aws sts get-caller-identity

# Lambda landscape
aws lambda list-functions --region ap-south-1 --max-items 20 \
  --query 'Functions[].{N:FunctionName,R:Runtime,T:Timeout,M:MemorySize}' --output table
aws lambda get-account-settings --region ap-south-1

# API Gateway (HTTP APIs)
aws apigatewayv2 get-apis --region ap-south-1 --query 'Items[].{N:Name,Id:ApiId,Ep:ProtocolType}' --output table

# Queues and topics
aws sqs list-queues --region ap-south-1
aws sns list-topics --region ap-south-1 --query 'Topics[].TopicArn'

# Event buses and state machines
aws events list-event-buses --region ap-south-1
aws stepfunctions list-state-machines --region ap-south-1 --query 'stateMachines[].name'

# DynamoDB tables (names only)
aws dynamodb list-tables --region ap-south-1

Sandbox + read-only IAM.

20. अजून industry आणि stack stories

• Fintech Lagos: webhook → SQS → idempotent ledger.
• Ecommerce Pune: throttle + fan-out.
• Healthcare SG: SNS + PHI IAM.
• EdTech: EventBridge cleanup.
• Logistics: bus + saga.
• Media: avoid recursive S3.
• SaaS: tenant partition.
• Gov: strangler peel.
• Startup: Function URL → HTTP API.
• Retail: FIFO vs Standard.

• Python: thin handlers.
• Node: async/await.
• Java: SnapStart.
• .NET: package size.
• Go: static binary.
• PHP: hybrid.

21. Glossary

22. Practice: मोठ्याने explain

1. 60s: queue load leveling गोष्ट.
2. 60s: sync vs async checkout.
3. 60s: Lambda नको कधी.
4. 60s: DLQ + idempotency.
5. Record ×2.

23. Safe lab hygiene

• Tag owner+expiry.
• Budget alarm आधी.
• Read-only first.
• Cleanup.
• No secrets in env/screenshots.
• No provisioned on hello.

24. या chapter च्या links

• Compute: Lambda.
• HTTP: API Gateway.
• Data: DynamoDB.
• Messaging: SQS/SNS.
• Events: EventBridge.
• Workflows: Step Functions.
• build-* projects link.

25. Myths

26. Serverless error budget

Checkout error budget 0.1% असेल तर aggressive retries email duplicates देऊन budget जाळू शकतात. Money SLO ≠ notify SLO.

1. दोन SLOs.
2. Money retries bounded.
3. Notify best-effort.
4. Money page; notify ticket.

27. Regional notes (ap-south-1)

• Labs Mumbai.
• Feature availability.
• Data residency.
• Latency India.

28. Naming conventions

# Suggested prefixes
# Lambda:  pay-intake-prod, pay-worker-prod
# SQS:     pay-orders.fifo / pay-orders-dlq
# SNS:     pay-order-events
# EB bus:  pay.domain
# SFN:     pay-refund-saga
# DDB:     pay-orders-prod

env + domain नावात ठेवा. CloudWatch grep सोपा.

29. Pre-mortems

1. Double charge headline.
2. DLQ drain plan.
3. Cognito outage mode.
4. Runbook.

30. Currency / locale

Currencies आणि names rotate करा — global mental model.

31. Failure timeline

1. Bad JSON throw.
2. Receive +1.
3. To DLQ.
4. Human inspect.
5. Fix+redrive.

32. Metrics

• Backlog.
• Age/SLA.
• Sent vs Deleted.
• Concurrency.

उजळणी, मुलाखत आणि तुलना cards

हा lesson काही मिनिटांत revise करा. Cards phone वर save करा किंवा PNG download करा. प्रत्येक card वरचा QR हा lesson उघडतो.

अजून cards: Ravi Sir's Cards.