Amazon SQS and SNS: Messaging Queues and Pub/Sub Explained
Your checkout API inserts an order and must email the customer, update inventory, and call fraud. If you do all three inside one request, one slow email makes shoppers wait. Amazon SQS queues work; Amazon SNS fans one event out to many subscribers — the async backbone of most AWS apps.
What you'll learn in this chapter
- Why async messaging beats tight coupling
- What SQS queues and SNS topics are
- How to design standard vs FIFO, DLQ, fan-out
- Industries using queues and pub/sub
- Lab, 8+ errors, FAQ, interview answers
1. Everyday example first
A Nagpur ecommerce site places order O-991. API writes the order to DynamoDB, then publishes OrderPlaced to SNS. Three SQS queues subscribe: email, inventory, fraud. If email workers are down, messages wait in the queue — checkout still returns 200.
2. Why SQS and SNS?
Why do we need this?
Monoliths call every side effect in one HTTP request. That couples latency and failures. Queues absorb spikes; topics broadcast. Together they give you retry, buffering, and independent scaling of workers.
- Decouple producers from slow consumers
- Buffer flash sales without losing orders
- Retry poison messages via DLQ instead of silent loss
- Fan-out one business event to many teams
3. What are SQS and SNS?
What is it?
SQS is a managed message queue: producers SendMessage, consumers ReceiveMessage then DeleteMessage. SNS is a pub/sub topic: Publish once, many subscriptions (SQS, Lambda, email, HTTP) get a copy. Most designs combine them: SNS fan-out into SQS for each worker type.
| Piece | Plain job |
|---|---|
| SQS standard | At-least-once, best-effort order, huge scale |
| SQS FIFO | Exactly-once processing helper + order per group |
| SNS topic | Fan-out publish to many subscribers |
| DLQ | Hold messages that fail repeatedly |
| Visibility timeout | Hide in-flight message while worker processes |
| Filter policy | SNS subscriber gets only matching attributes |
4. How to build a safe messaging path
How to do it (step by step)
- Create an SNS topic order-placed in ap-south-1.
- Create three SQS queues: email, inventory, fraud (start with standard).
- Subscribe each queue to the topic; set raw message delivery if you want pure JSON.
- Attach a DLQ to each queue with maxReceiveCount 3–5.
- Set visibility timeout ≥ longest processing time (often 30–120s).
- Grant the API role sns:Publish; grant workers sqs:ReceiveMessage/DeleteMessage.
- Idempotent workers: same message twice must not double-charge.
- Alarm on ApproximateNumberOfMessagesVisible and DLQ depth.
- For strict order per customer, use FIFO with MessageGroupId=customer_id.
- Load-test with a few hundred SendMessage calls before go-live.
# Mental model
# sns.publish(TopicArn=..., Message='{"order":"O-991"}')
# sqs.receive_message(QueueUrl=..., MaxNumberOfMessages=10, WaitTimeSeconds=20)
# process(); sqs.delete_message(QueueUrl=..., ReceiptHandle=...)5. SQS vs SNS vs EventBridge (quick)
| Choice | When |
|---|---|
| SQS | Work queue / competing consumers |
| SNS | Fan-out one event to many targets |
| EventBridge | Rich filters, SaaS partners, schedules |
| Kinesis | Ordered streams many consumers retain hours |
6. Multi-industry walkthrough
Banking posts ledger events to SNS; risk and notification queues subscribe with filter policies on amount tiers. A hospital lab system queues heavy PDF generation so the doctor UI stays fast. A SaaS billing service uses FIFO per tenant_id so invoice numbers never race.
7. Security checklist
- SSE encryption on queues and topics holding PII
- Least-privilege IAM on Publish/Receive/Delete
- No * Principal on queue policies
- Redact secrets from message bodies; use tokens
8. Industries and roles
- Ecommerce: order email and inventory
- Fintech: async KYC document checks
- EdTech: video transcoding jobs
- IT services: ticket fan-out to ops channels
- Healthcare: appointment reminders without blocking booking API
9. Lab: SNS fan-out to two SQS queues
Do this in ap-south-1 (Mumbai) unless your org forbids it. Set a budget alert first.
How to do it (step by step)
- Create topic lab-orders and queues lab-mail, lab-stock in ap-south-1.
- Subscribe both queues; publish a test JSON message.
- Receive from lab-mail; delete; confirm lab-stock still has a copy.
- Attach a DLQ to lab-mail with maxReceiveCount 3.
- Set a CloudWatch alarm on DLQ messages > 0.
Cleanup (stop billing)
- Delete subscriptions, queues, topic, alarms.
- Remove lab IAM users if any.
10. Common errors and fixes
Each row is a real mistake students and juniors hit. Read the fix before you panic.
| Error / symptom | Fix |
|---|---|
| Processed but forgot DeleteMessage | Always delete after success; tune visibility. |
| Visibility too short; duplicates explode | Set timeout > p99 process time. |
| Used FIFO without MessageGroupId plan | Group by business key (order/customer). |
| SNS to Lambda only; no DLQ story | Prefer SNS→SQS→Lambda for buffer+DLQ. |
| Hard-coded account IDs in policies | Use Conditions and wildcards carefully. |
| Poison JSON crashes worker forever | Validate; send to DLQ; alert. |
| Public queue policy by mistake | Principal only your account/roles. |
| Fan-out without filter; wasted work | SNS filter policies per subscriber. |
| At-least-once ignored; double email | Idempotency keys in DB. |
| No alarm on queue depth | Alarm + page on spike. |
11. Troubleshooting checklist
- Messages invisible: check visibility timeout and stuck workers.
- AccessDenied on Publish: topic policy + IAM.
- Duplicate deliveries: design idempotent handlers.
- FIFO throttles: more message groups or shard load.
- Long polling vs short: use WaitTimeSeconds 20.
- Encryption: enable SSE on queues/topics for PII.
12. FAQ
Q. SQS free?
Pay per request/data; free tier exists — check current pricing.
Q. Standard vs FIFO?
FIFO for order+dedup; standard for scale/throughput.
Q. SNS email for prod alerts?
OK for ops; app fan-out prefer SQS/Lambda.
Q. Replace Kafka with SQS?
Different model; Kafka for log streams.
Q. Exactly-once?
FIFO helps; still design idempotent side effects.
Q. Cross-region?
Queues are regional; replicate with patterns if needed.
13. Interview Q&A
Q. What is SQS?
Managed message queue for async work between producers and consumers.
Q. What is SNS?
Pub/sub topics that fan one message to many subscribers.
Q. Visibility timeout?
Time a message stays hidden after receive while you process it.
Q. Why DLQ?
Isolate poison messages after N fails so the main queue stays healthy.
Q. SNS+SQS pattern?
Topic fan-out into per-worker queues with buffer and DLQ.
Q. Idempotency?
Process the same delivery twice without double side effects.
14. Tips in English, Marathi and Hindi
Ravindra Bagale's Tip (EN)
Treat every SQS message as at-least-once. Build idempotent workers first; fancy FIFO later.
Ravindra Bagale's Tip (MR)
प्रत्येक SQS message = at-least-once. आधी idempotent workers; नंतर FIFO.
Tip (HI)
हर SQS संदेश को at-least-once मानो — पहले idempotent worker, फिर FIFO।
Try at home
- Draw SNS→3 SQS for a food-delivery order.
- Write an idempotent “mark emailed” design with a DynamoDB key.
- List three CloudWatch metrics you would alarm.
In short
- SQS queues work; SNS fans out events.
- Use DLQ + visibility + idempotency.
- Prefer SNS→SQS→worker for most app side effects.
- FIFO when order per group truly matters.
Revision, interview and comparison cards
Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.
More cards: Ravi Sir's Cards.
Practice Tasks
Maple Leaf Books is a bookselling business in Toronto. Emily Tremblay looks after its IT and cloud work. They count money in CAD and run their AWS resources in the ca-central-1 Region. In the tasks below you do real jobs for them using what this lesson teaches. All names, account IDs, IP addresses and numbers are made up for practice.
0 / 16
Easy List queues
Why: See messaging backlog.
What to do: List SQS queues.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Easy Get queue URL
Why: Need URL before send.
What to do: Get queue URL for mapleleafbooks-orders.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Easy List SNS topics
Why: Fan-out inventory.
What to do: List SNS topics.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Easy FIFO needs group — 1
Why: MessageGroupId.
What to do: Enter 1 if FIFO needs MessageGroupId.
- Write down the numbers the task gives you.
- Work it out step by step on paper.
- Type only the final answer in the box.
- Click Check answer.
Easy Delete after success — 1
Why: Ack pattern.
What to do: Enter 1 if you delete SQS message after success.
- Write down the numbers the task gives you.
- Work it out step by step on paper.
- Type only the final answer in the box.
- Click Check answer.
Medium Get queue attributes
Why: Check visibility/DLQ.
What to do: Get queue attributes All for https://sqs.ca-central-1.amazonaws.com/102932412000/mapleleafbooks-orders.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Medium Send message
Why: Produce work.
What to do: Send message hello to https://sqs.ca-central-1.amazonaws.com/102932412000/mapleleafbooks-orders.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Medium Receive messages
Why: Consume work.
What to do: Receive message on https://sqs.ca-central-1.amazonaws.com/102932412000/mapleleafbooks-orders.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Medium Get topic attributes
Why: Inspect topic.
What to do: Get topic attributes arn:aws:sns:ca-central-1:102932412000:mapleleafbooks-events.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Medium Allow Publish one topic
Why: Publisher LP.
What to do: Allow sns:Publish on arn:aws:sns:ca-central-1:102932412000:mapleleafbooks-events.
- Start with { "Version": "2012-10-17", "Statement": [ ... ] }.
- In each statement write Effect, Action and Resource.
- Give only what the task needs (least privilege).
- Use double quotes and commas correctly. Click Check answer.
Medium DLQ name JSON
Why: Name the DLQ.
What to do: JSON dlq_name orders-dlq.
- Start from the top-level keys the task names.
- Indent child keys with 2 spaces (no tabs).
- Fill in each property with the value from the task.
- Click Check answer. Key order does not matter.
Medium Prefer SNS to SQS buffer — 1
Why: Buffer+DLQ.
What to do: Enter 1 if SNS→SQS→Lambda is a durable pattern.
- Write down the numbers the task gives you.
- Work it out step by step on paper.
- Type only the final answer in the box.
- Click Check answer.
Medium Long poll seconds sample
Why: Cut empty receives.
What to do: Sample WaitTimeSeconds for long poll (enter 20)?
- Write down the numbers the task gives you.
- Work it out step by step on paper.
- Type only the final answer in the box.
- Click Check answer.
Hard Delete message
Why: Ack success.
What to do: Delete message on https://sqs.ca-central-1.amazonaws.com/102932412000/mapleleafbooks-orders receipt abc.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Hard Allow SendMessage one queue
Why: Producer least privilege.
What to do: Allow sqs:SendMessage on arn:aws:sqs:ca-central-1:102932412000:mapleleafbooks-orders.
- Start with { "Version": "2012-10-17", "Statement": [ ... ] }.
- In each statement write Effect, Action and Resource.
- Give only what the task needs (least privilege).
- Use double quotes and commas correctly. Click Check answer.
Hard Visibility timeout JSON
Why: Match process time.
What to do: JSON VisibilityTimeout 60.
- Start from the top-level keys the task names.
- Indent child keys with 2 spaces (no tabs).
- Fill in each property with the value from the task.
- Click Check answer. Key order does not matter.