Chapter 29: Serverless and Event-Driven Systems

29.1 Why.

Welcome to Chapter 29.

I am Ravindra Bagale.

Raju had no engineering degree. He had a function that ran when a message arrived, and a role attached to that function that could read every bucket in the lab account. He liked that there was no server to patch. Guru-ji liked that less.

"There is still a server," Guru-ji said. "You do not administer it. You still own the identity, the event, the dependency inside the package, and the log line. A queue is a door. If anyone can push a message, anyone can ask your function to work. If the function believes every message, you have built a remote control."

Raju pointed at the timeout and the cold start graph. "Suppose we are Swiggy and an order event fans out to five functions, and one of them has a role copied from a tutorial. Suppose we are Zomato and a retry storms the kitchen service because the function crashed after the write and before the acknowledgement. Suppose we are Paytm or PayFast and a payment event can be replayed, so a customer is charged twice or a ledger is edited twice. Suppose we are Amazon and a public function URL exists beside the private one. Suppose we are Netflix and a thumbnail function logs the full request, including a token. Suppose we are WhatsApp-shaped or BHIM UPI-shaped systems and the event carries more personal data than the consumer needs. Suppose we are Infosys and the client asks who can publish to the topic that starts production work. The absence of a virtual machine is not the absence of a threat model."

29.2 What.

Function identity. Each function gets its own role. The role can touch the one queue, the one table, and the one secret prefix that this function needs. It cannot administer the account. It cannot read other tenants. It cannot change its own role. Wildcards that mean "all resources" are a smell unless you can explain the blast in one sentence and still want it. Workload identity beats a long-lived key baked into the package. The package is copied, logged, and sometimes left in a build bucket. Keys do not belong there.

Event-source trust. Name who may produce the event. An API gateway with authentication is a different source from a bucket notification, and both are different from a queue that any principal in the account can publish to. Subscribe on purpose. Do not grant * on the send action because a demo did. If a partner sends webhooks, verify a signature the way Chapter 20 taught, and reject unsigned bodies. The function should not be reachable as an anonymous HTTP handler unless the product truly is public and the handler does almost nothing.

API gateways. The gateway is where authentication, throttling, and request size limits live. The function should not re-implement the whole edge badly, and it should not trust the gateway so much that it skips authorization on the data. A valid token tells you who is calling. An authorization check tells you whether that caller may touch this order. Keep both. Rate limits protect the budget from Chapter 27.

Secrets. Read secrets at runtime from a store, with the function role allowed to read one path. Do not compile secrets into the artifact. Do not print the environment on startup. Cold starts tempt people to log "just this once" to see why the function failed. That log is durable. Prefer a correlation id and a redacted error.

Dependencies and the package. A function artifact is a supply chain. Lock the versions. Scan the lockfile. Prefer a small package. An install script that runs at deploy time is code you did not review. Pin digests when you pull a base build image. Chapter 25's SLA applies to the libraries inside the zip, not only to hosts. Rebuild and redeploy when a library is in the emergency lane. A function you have not deployed in a year is a frozen vulnerability.

Events are data, not code. A message can contain quotes, scripts, or a string that looks like a command. Parse it as a schema: expected fields, expected types, maximum sizes. Reject the rest. Do not pass event text into a shell, a template engine, or a database by gluing strings. Do not turn a message into a permission decision without a server-side check. Injection through events is the serverless cousin of injection through forms. The defense is the same shape. This chapter does not include sample payloads that try to break a parser. Your lab test can send a too-large field and a missing field, and assert both are rejected.

Queues, topics, and dead letters. Authorize publishers and consumers separately. Encrypt the queue. Set a retention that matches the data class. A dead-letter queue catches messages that failed too many times so they do not loop forever and so you can inspect them without putting them back blindly. Access to the dead letter is sensitive, because failed messages may contain personal data. Alert on depth. A queue that only grows is an incident in slow motion.

Idempotency and replay. At-least-once delivery means your function will see the same event twice. Make the write safe to repeat. Store a processed-event id. Use a conditional update. Design the payment and order operations so a replay does not double-charge or double-book. Protect against a hostile replay too: a signature with a timestamp, or a nonce, on events from outside your account. Inside the account, least privilege on publish is the main brake. Test replay with a duplicate synthetic message you send yourself.

Observability. You do not have a host to log into, and that is fine. You do have invocation logs, traces, and metrics: errors, throttles, duration, and concurrent executions. Put a correlation id on the event and on the downstream call. Alarm on a sudden change in who invokes the function, not only on crashes. Cold start time is a reliability fact. It becomes a security fact if a timeout causes a retry that duplicates a payment. Tune timeout, retry, and idempotency together.

Infrastructure as code and policy. Define the function, the role, the queue, and the gateway in code. Scan that code for wildcards, public URLs, and missing dead-letter settings before apply. The golden template from Chapter 26 should grow a function path that already has a small role and a schema check. Drift still matters. A click in a console that widens the role should show up as a difference from Git.

A teaching sample: a role sketch. Lab only. The names are illustrative.

role: lab-order-fn
allow: read secret lab/order-fn/*
allow: send and receive on queue lab-orders
allow: write item on table lab-orders
deny: any action on * 
note: replace the star-deny with your cloud's explicit scope; do not ship a tutorial role

Seminar notes. Raju celebrated that there was no server to log into. Guru-ji said that was an operational convenience and a forensic constraint. You cannot repair what you cannot see, and you cannot see a function you did not instrument. Cold start is the delay before the first invocation warms. It is a reliability fact. It becomes a security fact when a timeout causes a retry and the retry repeats a payment-shaped write. Design idempotency first, then tune the timeout, then set the retry count, then send the exhausted message to a dead letter a human can read under an authorized role. Do not log the full event if the event holds personal data. Log the id, the result, and the correlation id. WhatsApp-shaped payloads and BHIM UPI-shaped payloads do not belong in an error string.

Event-source trust is a list you can draw. Who can invoke the function by URL. Who can publish to the topic. Who can put an object that triggers a run. Each publisher is an entry point, the way an API route is an entry point. A forgotten URL with no authentication is a public handler. Delete it or put the gateway in front with authentication and a size limit. The function still checks authorization on the data. Gateway authentication is not object authorization. Swiggy-shaped order events should name the publisher role in the repository so a new subscription is a reviewed diff, not a click that nobody repeats in Git.

Infrastructure scanning for serverless is the same policy habit as Chapter 16, aimed at a different resource. Fail a lab apply that uses a wildcard resource, a public function URL, a missing dead-letter setting, or a role that can administer identities. The golden template should already pass that scan so the paved function is the easy function. Drift is a console edit. Detect it. Supply-chain controls are the lockfile, the pinned build image, and a role so small that a bad library has little cloud power. You still patch by rebuilding. There is no host package manager to save you if you never redeploy. Chapter 25's age metric applies to the running digest of the function, not to the repository's intentions.

Queues need encryption, retention that matches the data class, and separate publish and consume rights. A dead-letter queue is a second store of the same data. Lock it. Alert on its depth so failure is visible before someone pastes messages back into production blindly. Replay protection for outside senders is a signature and a timestamp checked in your code. For inside senders, the main control is who may publish. Test both with synthetic events you create. Never send those events to a company endpoint to see if their function is careful. Their function is not your lab.

Observability without a server still has a bill and a blast radius. Reserved concurrency is a ceiling. Use one in the lab and watch the calm throttle. Traces should cross the gateway, the function, and the table write so an incident has a path. If the only debug tool is printing the environment, remove that tool. Secrets in logs are a data incident you caused to save ten minutes. Paytm-shaped and PayFast-shaped functions that settle money should have the strictest ceilings and the strictest logs. A thumbnail-style function can be looser and must still not hold a role that can settle money. Split them. One artifact, one role, one purpose.

One more pass. Guru-ji asked Raju to price a bug. A public function that writes orders is not a small bug. A private function with a huge role is not a small bug either. Write the price next to the resource in Git so the scanner's severity is not the only opinion. API gateway integration should make the price visible: authentication method, throttle, maximum body size, and which routes exist. A route you forgot is still a route. List them in the catalog. Idempotency keys belong in the client contract for any write that money or inventory depends on. The lab proves a duplicate does not double-apply. It does not prove this by hammering a real payment API. Synthetic table, synthetic key, one row.

Cold starts also change logs. The first call may initialize a client and fail differently from the warm call. Test both so you do not "fix" a failure by printing secrets only on the cold path. Package size is a supply-chain choice. Smaller packages have fewer libraries to urgent-patch. Rebuild on the emergency lane the same day you would patch a host, because the function will not reboot itself into a new library. IaC is the deploy. If the IaC and the live role disagree, believe the live role until you reconcile, then believe Git. That is the same drift rule the next chapters will make mechanical.

A short close on the package. The function artifact should be rebuildable from a lockfile and a pinned builder. If you cannot rebuild it, you cannot patch it. Write the rebuild command in the catalog row. When a known-exploited library appears, the emergency lane is a rebuild, a scan, a small canary, and a full promote. There is no in-place patch on a running interpreter you do not control. That is the bargain of this chapter. You trade host maintenance for artifact discipline. Keep the discipline.

29.3 How.

1. Write "own lab only" on the design page for a tiny function.
2. Define one synthetic event schema: three fields, types, and maximum lengths.
3. Create a queue or topic that only your lab publisher role can write.
4. Create a function role that can read that queue and write one table, and nothing else.
5. Confirm the role cannot list other buckets or change users. Record the deny.
6. Store a fake secret in a lab secret path and allow only this role to read it.
7. Build the package from a lockfile and run a dependency scan in the pipeline.
8. Fail the build on a test dependency you marked as forbidden, then remove it.
9. Deploy through infrastructure code, not through a console click you will forget.
10. Send a valid synthetic event and confirm one row is written.
11. Send the same event again and confirm you still have one row.
12. Send an event with an extra huge field and confirm it is rejected and not executed as a command.
13. Point failures at a dead-letter queue after a small retry count.
14. Restrict who can read the dead letter.
15. Turn off verbose environment dumps in the log configuration.
16. Add a correlation id and find one trace that crosses the function and the table write.
17. Set a reserved concurrency or a modest quota so a loop cannot scale without a ceiling.
18. Add an alarm on error rate and on dead-letter depth.
19. Change the role in Git, apply, and show the catalog or the diff.
20. Revert a console change on purpose and see your drift check catch it.
21. Document publishers, consumers, data class, and the idempotency key.
22. Stop. Do not fuzz, invoke, or subscribe to systems at Swiggy, Paytm, Amazon, BHIM UPI, WhatsApp, Zomato, Netflix, Infosys, PayFast, or any account you do not own.

29.4 A real public incident.

Name. Compromise of the ua-parser-js package on the npm registry.

Date. Malicious versions were published on 22 October 2021 after the maintainer's account was taken over. Advisories and the maintainer's public notices followed the same day.

Source. The GitHub security advisory for ua-parser-js and the maintainer's public statements on 22 October 2021. The package is widely embedded in services and in function artifacts. This chapter does not describe what the malicious releases executed, and it does not list commands.

What happened. A popular library, the kind a function bundles without a second thought, was published under its real name with versions the maintainer did not intend. Anyone who installed or upgraded into those versions during the window pulled code they did not review. That is a supply-chain event aimed at the artifact, not at a host you can patch with a system package. Serverless makes the lesson sharper. The artifact is the deploy. A lockfile that floats, a pipeline that auto-upgrades, or a role that can do anything once the process runs, turns a bad publish into a broad cloud action. The defensive response published at the time was to avoid the bad versions, use a known good version, and rotate credentials that the install could have seen. Detection depended on people noticing the unexpected publish and on advisories traveling fast.

How to stop.

1. Pin deployments to known good versions and redeploy functions that may have resolved the bad versions.
2. Rotate secrets and tokens that those functions or build jobs could access.
3. Review logs for unexpected use of the function role during the window.
4. Preserve the build logs. Do not keep a copy of the malicious package as a toy.
5. Tell owners of downstream artifacts.

How to prevent it again.

1. Use lockfiles and private registries or allow-lists for production functions.
2. Scan artifacts in the pipeline and fail closed on forbidden versions.
3. Keep function roles small so a bad library cannot act as an account admin.
4. Separate build credentials from production credentials.
5. Alert on dependency changes the way you alert on code changes.
6. Practice the pin and the rotate in a lab you own. Do not publish packages to harm others, and do not fetch "samples" of malware to run.

This is for learning only, and only inside a system you are allowed to test.

29.5 A simple picture.

An event is allowed in only from a named publisher. The function role can do one job. A duplicate event meets an idempotency key and stops.

29.6 A short animation.

A duplicate event blinks, the idempotency key holds, and the table stays at one row.

29.7 Conclusion.

Raju cut the function role until the policy was boring. The duplicate event did nothing the second time. Guru-ji looked pleased. "In Chapter 30 the client is a phone you do not control. The store on that phone is hostile ground. The trust stays on your API, and the signing key stays in the pipeline."

• A function still has an identity. Make it small.
• Events are data. Schema-check them. Do not turn them into commands.
• Replays happen. Idempotency is a security control for payments and orders.
• The ua-parser-js publishes of 22 October 2021 are the frozen-artifact lesson.
• Own lab only.

END OF CHAPTER 29 MASTER CLASS