Fan-Out and Fan-In Patterns with SNS, SQS and Step Functions

By Ravindra Bagale, trainer · LinkedIn · Last updated: 2 November 2026

1. Everyday example: school bell and sports board

Rohan in Pune rings one SNS topic OrderPlaced; inventory, invoice (INR) and analytics each get a copy. Later, three fraud checks must all finish before ship — Step Functions Parallel then join. That join is fan-in.

Hands-on: order fanout project. Messaging depth: SQS/SNS. Orchestration depth: Step Functions.

2. Why fan-out?

Why do we need this?

Calling three Lambdas sync from one handler couples timeouts and permissions. Fan-out lets each consumer scale, fail and retry alone.

What is it?

Fan-out: one signal → many consumers. Fan-in: many branches complete → one next step.

3. Vertical diagrams

4. SNS vs EventBridge vs SFN Parallel

5. How: safe SNS → SQS → Lambda fan-out

How to do it (step by step)

1. Create topic shop-order-placed.
2. Create three queues + DLQs; subscribe queues to topic.
3. Raw message delivery off/on consciously (JSON envelope).
4. Event source mapping from each queue to its Lambda.
5. IAM: producer Publish; consumers Consume+DLQ.
6. Publish one test; confirm three processing paths.
7. Alarm on each DLQ depth.

List topic subscriptions

aws sns list-subscriptions-by-topic --topic-arn $TOPIC --region ap-south-1

Meaning: Confirm three SQS subscriptions before load test.

Publish a test JSON

aws sns publish --topic-arn $TOPIC --region ap-south-1 \
  --message '{"orderId":"o-42","currency":"INR","total":499}'

Meaning: Watch three queues receive a copy (ApproximateNumberOfMessages).

6. Fan-in patterns

• Step Functions Parallel: fixed branches, automatic join.
• Map state: dynamic array, capped concurrency, then continue.
• DynamoDB counter/condition: advanced; easy to get wrong — prefer SFN.
• Never busy-poll three APIs from a user-facing Lambda without timeout math.

Python: decide buffer or direct

def needs_queue(consumer_slow, spike_expected):
    return consumer_slow or spike_expected
print(needs_queue(True, False))
print(needs_queue(False, False))

Output: True / False. Meaning: slow or spiky consumers get SQS.

7. Failure isolation

If invoice Lambda fails, inventory must still succeed. That is why you fan-out through a topic/bus instead of a for-loop of sync invokes. Luis in Mexico City learned this when MXN invoicing outage blocked stock reserve.

8. Lab: paper fan-out + optional SNS lab

Do this in ap-south-1 (Mumbai) unless your org forbids it. Set a budget alert first.

How to do it (step by step)

1. Draw OrderPlaced → three consumers with DLQs.
2. Mark which consumer needs FIFO vs Standard.
3. Optional: create topic+2 queues subscription in sandbox.
4. Publish once; verify two receives.
5. Delete topic, queues, subscriptions.

Cleanup (stop billing)

1. Unsubscribe then delete queues/topic.
2. Remove lab Lambdas/log groups.

9. Map state concurrency

Map over 10,000 items with concurrency 1,000 can melt downstream. Set MaxConcurrency; prefer SQS between Map and fragile APIs.

Approximate queue depth

aws sqs get-queue-attributes --queue-url $Q --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible --region ap-south-1

Meaning: Depth rising while API healthy → consumer lag, not producer loss.

10. Idempotency under fan-out

SNS/SQS can deliver duplicates. Consumers must treat orderId as idempotent key (DynamoDB conditional put). Kenji in Osaka double-sent JPY invoices until the condition key landed.

11. Same pattern across industries

• Fintech (Aisha, Lagos): payment webhooks → SQS → Lambda; never block the bank callback thread.
• Ecommerce (Rohan, Pune): checkout API on API Gateway+Lambda; inventory fan-out on SNS.
• Healthcare (Mei, Singapore): appointment SMS via SNS; PHI stays in DynamoDB with fine-grained IAM.
• EdTech (Diego, São Paulo): nightly cleanup EventBridge → Lambda on student drafts.
• Logistics (Fatima, Casablanca): package scan events on EventBridge bus; Step Functions for customs saga.
• Media (Kenji, Osaka): S3 upload → Lambda thumbnails; fan-out to moderation + CDN invalidate.
• SaaS multi-tenant (Priya, Bengaluru): Cognito groups + DynamoDB partition per tenant.
• Gov portal (Omar, Amman): strangler fig — peel one PDF form route per sprint to Lambda.

12. Troubleshooting checklist

1. Confirm region (ap-south-1 vs us-east-1 for CloudFront/ACM quirks).
2. Check CloudWatch Logs for the function name and request id.
3. Verify IAM execution role allows the API you call.
4. For async: check DLQ depth and SQS age of oldest message.
5. For APIs: curl -v and compare status, CORS headers, body.
6. For workflows: open Step Functions execution event history.
7. Re-read the linked deep chapter before reinventing a service.
8. If cost spikes: check concurrency, retries, and provisioned capacity.

13. Common errors and fixes

Each row is a real mistake students and juniors hit. Read the fix before you panic.

14. FAQ

15. Interview Q&A (Beginner / Mid / Senior + MNC)

16. Tips in English, Marathi and Hindi

17. Poison and redrive under fan-out

1. Each consumer queue has its own DLQ.
2. Alarm text names the consumer team.
3. Fix code; redrive only that DLQ.
4. Never redrive blindly during an outage without a sample inspect.

18. Security notes

• Topic policy: who can Publish.
• Queue policy: allow SNS source ARN only.
• Lambda role: one queue consume, not *.
• Encrypt queues with SSE (SQS managed or KMS).

19. Cost notes

SNS + SQS + Lambda for three consumers costs more than one sync Lambda — you buy isolation. At low volume free tier often covers labs. At Black Friday INR scale, watch SQS request units and Lambda concurrency.

Filter policy example (concept)

# Subscription attribute FilterPolicyJSON
# {"currency":["INR","AED"],"type":["OrderPlaced"]}

Meaning: Consumer wakes only for matching currencies.

20. Architecture decision checklist

1. Write the user journey in five bullets with no AWS names.
2. Mark each step sync (user waits) or async.
3. Name failure mode: retry, DLQ, compensate, or alert-only.
4. Name the idempotency key for every write.
5. Pick data store from access patterns (link DynamoDB modeling).
6. Estimate monthly cost at 1x and 10x traffic (USD and INR).
7. Confirm account concurrency and service quotas in ap-south-1.
8. Decide auth: Cognito, IAM, Lambda authorizer, or none.
9. Name on-call and which CloudWatch alarm pages them.
10. Link the deep chapter you will not rewrite.
11. Only then open console or IaC.

21. Copy-paste quick commands (read-only first)

# Region and identity
aws configure get region
aws sts get-caller-identity

# Lambda landscape
aws lambda list-functions --region ap-south-1 --max-items 20 \
  --query 'Functions[].{N:FunctionName,R:Runtime,T:Timeout,M:MemorySize}' --output table
aws lambda get-account-settings --region ap-south-1

# API Gateway (HTTP APIs)
aws apigatewayv2 get-apis --region ap-south-1 --query 'Items[].{N:Name,Id:ApiId,Ep:ProtocolType}' --output table

# Queues and topics
aws sqs list-queues --region ap-south-1
aws sns list-topics --region ap-south-1 --query 'Topics[].TopicArn'

# Event buses and state machines
aws events list-event-buses --region ap-south-1
aws stepfunctions list-state-machines --region ap-south-1 --query 'stateMachines[].name'

# DynamoDB tables (names only)
aws dynamodb list-tables --region ap-south-1

Run these in a sandbox account. Prefer read-only IAM while learning.

22. More industry and stack stories

• Fintech (Aisha, Lagos, NGN): bank webhooks must return 200 fast — SQS absorbs spikes; ledger Lambda is idempotent on paymentId.
• Ecommerce (Rohan, Pune, INR): Black Friday — API Gateway throttle + SQS order intake; SNS fans inventory, email, analytics.
• Healthcare (Mei, Singapore, SGD): appointment reminders via SNS SMS; PHI only in DynamoDB with condition keys.
• EdTech (Diego, São Paulo, BRL): EventBridge nightly cleanup of draft quizzes; Lambda timeout 30s, not 15 min.
• Logistics (Fatima, Casablanca, MAD): package scans on custom EventBridge bus; Step Functions customs saga.
• Media (Kenji, Osaka, JPY): S3 → Lambda thumbnails; never write thumbs to the watched prefix.
• SaaS (Priya, Bengaluru, INR): Cognito groups + DynamoDB PK tenantId; one Lambda authorizer for all routes.
• Gov (Omar, Amman, JOD): strangler — peel one PDF form route per sprint from the monolith.
• Startup (Sofia, Sofia, EUR): start with Function URL for demos; move to HTTP API when partners need JWT.
• Retail (Luis, Mexico City, MXN): store POS sync via SQS FIFO when order matters; Standard when it does not.

• Python: handlers thin; put domain logic in plain modules for unit tests without Lambda.
• Node: prefer async/await; never block the event loop on sync DNS.
• Java: SnapStart for cold starts; keep dependencies lean.
• .NET: AspNetCoreServer for low-traffic tenants; watch package size.
• Go: single static binary; great for SQS workers.
• PHP: keep Laravel queues on ECS; Lambda for S3/webhooks glue.

23. Glossary (short)

24. Practice: explain out loud

1. 60 seconds: everyday story for fan-out and fan-in.
2. 60 seconds: sync vs async with a checkout example.
3. 60 seconds: when you refuse Lambda.
4. 60 seconds: DLQ + idempotency in one breath.
5. Record once; cut fillers; record again.

25. Safe lab hygiene

• Tag every lab resource with owner and expiry date.
• Set a monthly AWS Budget alarm in INR or USD before experiments.
• Prefer read-only commands first; mutate in a sandbox account.
• Delete queues, APIs, tables you created — DynamoDB and APIGW leave footprints.
• Never put production secrets in env vars or screenshots.
• Do not enable provisioned concurrency for a hello-world.

26. How this chapter links others

• Compute depth: Lambda.
• HTTP edge: API Gateway.
• Data modeling: DynamoDB.
• Messaging: SQS/SNS.
• Events: EventBridge.
• Workflows: Step Functions.
• Project chapters (build-*): worked assemblies — link, do not paste.

27. Myths to drop

28. Error budget thinking for serverless

If your monthly error budget is 0.1% on checkout, a fan-out that retries aggressively can burn the budget with duplicate emails while money stays correct. Separate business correctness SLOs from notification SLOs.

1. Write two SLOs: money-correct and notify-delivered.
2. Retries on money path: bounded + idempotent.
3. Retries on notify path: best-effort + DLQ.
4. Page humans on money DLQ; ticket on notify DLQ.

29. Regional notes (ap-south-1)

• Build labs in Mumbai (ap-south-1) unless your org mandates elsewhere.
• Some services launch first in us-east-1 — check feature availability.
• Data residency for INR fintech may require in-region only.
• Latency to end users in India usually favors ap-south-1.

30. Naming conventions that save incidents

# Suggested prefixes
# Lambda:  pay-intake-prod, pay-worker-prod
# SQS:     pay-orders.fifo / pay-orders-dlq
# SNS:     pay-order-events
# EB bus:  pay.domain
# SFN:     pay-refund-saga
# DDB:     pay-orders-prod

Include env (dev/stage/prod) and domain. Future you grepping CloudWatch will thank present you.

31. Pre-mortems (ten minutes)

1. Imagine tomorrow headline: double charge in AED. What broke?
2. Imagine DLQ depth 50,000 after a bad deploy. How do you drain?
3. Imagine Cognito outage. Do APIs fail closed or serve read-only?
4. Write answers in the runbook before go-live.

32. Currency and locale in examples

Rotate currencies in designs: INR, USD, EUR, AED, NGN, BRL, JPY, SGD. Rotate names: Aisha, Rohan, Mei, Diego, Fatima, Kenji, Priya, Omar, Sofia, Luis. This keeps mental models global — not one-country samples.

Revision, interview and comparison cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

More cards: Ravi Sir's Cards.