Serverless Observability: X-Ray, Powertools and Structured Logs

By Ravindra Bagale, trainer · LinkedIn · Last updated: 2 November 2026

1. Everyday example: courier tracking number

Fatima in Casablanca (MAD) pastes a tracking id and sees each warehouse scan. Diego in São Paulo (BRL) needs the same for checkout: API Gateway request id → Lambda → DynamoDB. Without one id in every log line, he greps in the dark during a Black Friday incident.

Metrics/alarms depth: CloudWatch. Compute basics: Lambda. Here we wire an observability pattern for serverless apps.

2. Why observability pattern?

Why do we need this?

No SSH. Short-lived environments. Failures hide in retries and async hops. You need logs, metrics, traces, and alarms designed on day one — not after the first 2 AM page in USD production.

What is it?

Serverless observability = structured logs + metrics (EMF/Powertools) + distributed traces (X-Ray) + actionable alarms, all correlated by ids (requestId, orderId, trace id).

3. Vertical diagram

4. How: minimum viable observe

How to do it (step by step)

1. JSON log each request with level, msg, orderId, requestId.
2. Enable X-Ray on API stage + Lambda; use AWS SDK instrumentation.
3. Emit 3 business metrics: OrdersCreated, PaymentFailed, LagMs.
4. Alarm: errors, throttles, DLQ depth, iterator age, p99 latency.
5. Alarm description = runbook URL + owner.
6. Redact PII — never log full card or national id.

5. Per-concept commands

Tail recent Lambda logs

aws logs tail /aws/lambda/checkout-api --region ap-south-1 --since 10m --format short

Meaning: Fast incident view; prefer JSON fields you can filter.

X-Ray service graph summary

aws xray get-service-graph --region ap-south-1 \
  --start-time $(date -u -d '30 minutes ago' +%s) \
  --end-time $(date -u +%s) \
  --query 'Services[].{N:Name,E:RootSummaryStatistics.ErrorCount}' 

Meaning: Shows which hop is erroring in the last half hour.

Put a safe test metric alarm sketch

aws cloudwatch describe-alarms --region ap-south-1 \
  --alarm-name-prefix checkout --max-records 10 \
  --query 'MetricAlarms[].{N:AlarmName,S:StateValue}' 

Meaning: Lists checkout alarms and state — verify WARNING/ALARM paths in staging first.

Filter logs by orderId

aws logs filter-log-events --region ap-south-1 \
  --log-group-name /aws/lambda/checkout-api \
  --filter-pattern '{ $.orderId = "ord-7841" }' --limit 20 

Meaning: Correlation search — only works if you logged orderId as JSON.

6. Powertools habits

• Logger: bind correlation keys once per invoke.
• Tracer: capture outbound SDK calls.
• Metrics: cold start metric dimension.
• Idempotency utility where payments matter.

7. Alarm design

8. Industries

• Fintech (Aisha, Lagos, NGN): ledger correlation mandatory.
• Ecommerce (Rohan, Pune, INR): checkout trace + payment metric.
• Healthcare (Mei, Singapore, SGD): redact PHI in logs.
• Logistics (Omar, Amman, JOD): scan events metrics per hub.
• SaaS (Priya, Bengaluru, INR): per-tenant error dashboards.

9. Lab

10. Lab - logs + one alarm

Do this in ap-south-1 (Mumbai) unless your org forbids it. Set a budget alert first.

How to do it (step by step)

1. Invoke a lab Lambda; confirm JSON log with a fake orderId.
2. Enable X-Ray on that function; open service map once.
3. Create or inspect an alarm on Errors with SNS email in sandbox.
4. Write a 5-line runbook in a text file for that alarm.

Cleanup (stop billing)

1. Delete lab alarms and SNS test subscriptions.
2. Disable X-Ray if not needed to save cost.

11. Same pattern across industries

• Fintech (Aisha, Lagos): payment webhooks → SQS → Lambda; never block the bank callback thread.
• Ecommerce (Rohan, Pune): checkout API on API Gateway+Lambda; inventory fan-out on SNS.
• Healthcare (Mei, Singapore): appointment SMS via SNS; PHI stays in DynamoDB with fine-grained IAM.
• EdTech (Diego, São Paulo): nightly cleanup EventBridge → Lambda on student drafts.
• Logistics (Fatima, Casablanca): package scan events on EventBridge bus; Step Functions for customs saga.
• Media (Kenji, Osaka): S3 upload → Lambda thumbnails; fan-out to moderation + CDN invalidate.
• SaaS multi-tenant (Priya, Bengaluru): Cognito groups + DynamoDB partition per tenant.
• Gov portal (Omar, Amman): strangler fig — peel one PDF form route per sprint to Lambda.

12. Troubleshooting checklist

1. Confirm region (ap-south-1 vs us-east-1 for CloudFront/ACM quirks).
2. Check CloudWatch Logs for the function name and request id.
3. Verify IAM execution role allows the API you call.
4. For async: check DLQ depth and SQS age of oldest message.
5. For APIs: curl -v and compare status, CORS headers, body.
6. For workflows: open Step Functions execution event history.
7. Re-read the linked deep chapter before reinventing a service.
8. If cost spikes: check concurrency, retries, and provisioned capacity.

13. Common errors and fixes

Each row is a real mistake students and juniors hit. Read the fix before you panic.

14. FAQ

15. Interview Q&A (Beginner / Mid / Senior + MNC)

16. Tips in English, Marathi and Hindi

17. Architecture decision checklist

1. Write the user journey in five bullets with no AWS names.
2. Mark each step sync (user waits) or async.
3. Name failure mode: retry, DLQ, compensate, or alert-only.
4. Name the idempotency key for every write.
5. Pick data store from access patterns (link DynamoDB modeling).
6. Estimate monthly cost at 1x and 10x traffic (USD and INR).
7. Confirm account concurrency and service quotas in ap-south-1.
8. Decide auth: Cognito, IAM, Lambda authorizer, or none.
9. Name on-call and which CloudWatch alarm pages them.
10. Link the deep chapter you will not rewrite.
11. Only then open console or IaC.

18. Copy-paste quick commands (read-only first)

# Region and identity
aws configure get region
aws sts get-caller-identity

# Lambda landscape
aws lambda list-functions --region ap-south-1 --max-items 20 \
  --query 'Functions[].{N:FunctionName,R:Runtime,T:Timeout,M:MemorySize}' --output table
aws lambda get-account-settings --region ap-south-1

# API Gateway (HTTP APIs)
aws apigatewayv2 get-apis --region ap-south-1 --query 'Items[].{N:Name,Id:ApiId,Ep:ProtocolType}' --output table

# Queues and topics
aws sqs list-queues --region ap-south-1
aws sns list-topics --region ap-south-1 --query 'Topics[].TopicArn'

# Event buses and state machines
aws events list-event-buses --region ap-south-1
aws stepfunctions list-state-machines --region ap-south-1 --query 'stateMachines[].name'

# DynamoDB tables (names only)
aws dynamodb list-tables --region ap-south-1

Run these in a sandbox account. Prefer read-only IAM while learning.

19. More industry and stack stories

• Fintech (Aisha, Lagos, NGN): bank webhooks must return 200 fast — SQS absorbs spikes; ledger Lambda is idempotent on paymentId.
• Ecommerce (Rohan, Pune, INR): Black Friday — API Gateway throttle + SQS order intake; SNS fans inventory, email, analytics.
• Healthcare (Mei, Singapore, SGD): appointment reminders via SNS SMS; PHI only in DynamoDB with condition keys.
• EdTech (Diego, São Paulo, BRL): EventBridge nightly cleanup of draft quizzes; Lambda timeout 30s, not 15 min.
• Logistics (Fatima, Casablanca, MAD): package scans on custom EventBridge bus; Step Functions customs saga.
• Media (Kenji, Osaka, JPY): S3 → Lambda thumbnails; never write thumbs to the watched prefix.
• SaaS (Priya, Bengaluru, INR): Cognito groups + DynamoDB PK tenantId; one Lambda authorizer for all routes.
• Gov (Omar, Amman, JOD): strangler — peel one PDF form route per sprint from the monolith.
• Startup (Sofia, Sofia, EUR): start with Function URL for demos; move to HTTP API when partners need JWT.
• Retail (Luis, Mexico City, MXN): store POS sync via SQS FIFO when order matters; Standard when it does not.

• Python: handlers thin; put domain logic in plain modules for unit tests without Lambda.
• Node: prefer async/await; never block the event loop on sync DNS.
• Java: SnapStart for cold starts; keep dependencies lean.
• .NET: AspNetCoreServer for low-traffic tenants; watch package size.
• Go: single static binary; great for SQS workers.
• PHP: keep Laravel queues on ECS; Lambda for S3/webhooks glue.

20. Glossary (short)

21. Practice: explain out loud

1. 60 seconds: everyday story for serverless observability.
2. 60 seconds: sync vs async with a checkout example.
3. 60 seconds: when you refuse Lambda.
4. 60 seconds: DLQ + idempotency in one breath.
5. Record once; cut fillers; record again.

22. Safe lab hygiene

• Tag every lab resource with owner and expiry date.
• Set a monthly AWS Budget alarm in INR or USD before experiments.
• Prefer read-only commands first; mutate in a sandbox account.
• Delete queues, APIs, tables you created — DynamoDB and APIGW leave footprints.
• Never put production secrets in env vars or screenshots.
• Do not enable provisioned concurrency for a hello-world.

23. How this chapter links others

• Compute depth: Lambda.
• HTTP edge: API Gateway.
• Data modeling: DynamoDB.
• Messaging: SQS/SNS.
• Events: EventBridge.
• Workflows: Step Functions.
• Project chapters (build-*): worked assemblies — link, do not paste.

24. Myths to drop

25. Error budget thinking for serverless

If your monthly error budget is 0.1% on checkout, a fan-out that retries aggressively can burn the budget with duplicate emails while money stays correct. Separate business correctness SLOs from notification SLOs.

1. Write two SLOs: money-correct and notify-delivered.
2. Retries on money path: bounded + idempotent.
3. Retries on notify path: best-effort + DLQ.
4. Page humans on money DLQ; ticket on notify DLQ.

26. Regional notes (ap-south-1)

• Build labs in Mumbai (ap-south-1) unless your org mandates elsewhere.
• Some services launch first in us-east-1 — check feature availability.
• Data residency for INR fintech may require in-region only.
• Latency to end users in India usually favors ap-south-1.

27. Naming conventions that save incidents

# Suggested prefixes
# Lambda:  pay-intake-prod, pay-worker-prod
# SQS:     pay-orders.fifo / pay-orders-dlq
# SNS:     pay-order-events
# EB bus:  pay.domain
# SFN:     pay-refund-saga
# DDB:     pay-orders-prod

Include env (dev/stage/prod) and domain. Future you grepping CloudWatch will thank present you.

28. Pre-mortems (ten minutes)

1. Imagine tomorrow headline: double charge in AED. What broke?
2. Imagine DLQ depth 50,000 after a bad deploy. How do you drain?
3. Imagine Cognito outage. Do APIs fail closed or serve read-only?
4. Write answers in the runbook before go-live.

29. Currency and locale in examples

Rotate currencies in designs: INR, USD, EUR, AED, NGN, BRL, JPY, SGD. Rotate names: Aisha, Rohan, Mei, Diego, Fatima, Kenji, Priya, Omar, Sofia, Luis. This keeps mental models global — not one-country samples.

30. Worked story: Aisha ledger page (NGN)

1. PaymentFailed metric spikes; alarm pages.
2. X-Ray shows DynamoDB throttle on ledger table.
3. Logs filter paymentId show retries exhausted to DLQ.
4. Fix capacity; redrive; postmortem updates runbook.

31. Anti-patterns

• console.log without request id.
• Alarm with no owner or runbook.
• Tracing 100% including synthetic freeloaders.

Revision, interview and comparison cards

Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.

More cards: Ravi Sir's Cards.