Chapter 20: API, Microservices, and Service-to-Service Security

20.1 Why.

Welcome to Chapter 20.

I am Ravindra Bagale.

Raju had no engineering degree. He had a route called /orders/1 and a feeling that hiding the link was a lock. Guru-ji asked for order 2. The lab returned it. "The number in the path is not a secret," Guru-ji said. "It is an address. Security is the check that this caller may open this address, and the check that the service behind the address may speak to the next service as itself, not as a shared password."

Raju remembered Chapter 19's session. "The user already logged in."

"Logged in to the lobby," Guru-ji said. "The kitchen is a different room. Suppose we are Swiggy and a customer token can read every restaurant's orders by changing an id. Suppose we are Zomato and a partner webhook accepts any body that looks like JSON. Suppose we are Paytm or PayFast and a payment API returns the full card-shaped record because the mobile screen only shows the last digits. Suppose we are BHIM UPI and a replay of a signed request moves money twice because nobody stored the idempotency key. Suppose we are Amazon and a forgotten internal API still sits on the internet without an owner. Suppose we are Netflix and service A calls service B with a token that was meant for service C. Suppose we are WhatsApp-shaped messaging and a GraphQL query asks for a nest so deep the database kneels. Suppose we are Infosys delivering a client platform and the inventory of routes is a rumor. APIs are the city streets. Unowned streets become the incident."

20.2 What.

REST, GraphQL, and gRPC. REST is resources and verbs over HTTP. GraphQL is a query language where the client chooses the shape. gRPC is a contract of messages, often on HTTP/2, common between services inside a cluster. The security idea does not change with the style. Authenticate the caller. Authorize the object. Validate the input. Limit the rate. Log the decision. Inventory the method. A new style does not retire an old bug. It only changes the spelling.

API keys, OAuth, OIDC, and JWT. An API key is a shared secret that names a client. It is acceptable for a low-risk machine call you can rotate, and it is a poor stand-in for a user. Keys leak into mobile apps, tickets, and logs. Prefer OAuth 2 and OpenID Connect when a person is involved, which Chapter 19 introduced. The user authenticates at the identity provider. Your API receives a token. A JWT is one token format: a signed set of claims. Validate it. Do not decode it and trust the middle because the middle is readable. Readable is not true.

Claims you must check. Issuer: who signed this. Audience: which API it was minted for. Expiry and not-before: whether it is alive. Subject: which principal. Algorithm: the one you pinned, never "whatever the token header asks." A token for the menu service must not be accepted by the payment service. A token from a lab issuer must not be accepted by a second lab that forgot to change the default. Clock skew of a few seconds is normal. A month of skew is a bug. Rotate verification keys. Reject none and reject algorithms you do not use. This chapter does not show how to mint a forged token. It shows the checks that make forgery fail closed.

A lab validator, in words:

if issuer != LAB_ISSUER: deny
if audience != LAB_ORDERS_API: deny
if now > expiry: deny
if algorithm != RS256: deny
if signature fails: deny
if subject has no right to this order id: deny

The last line is authorization. The lines above are authentication of the token. You need both.

BOLA, BFLA, mass assignment, and excessive data. Broken object level authorization means the caller is real and the object is not theirs. Order 2 opened because the code checked the token and not the owner. Broken function level authorization means a customer can call an admin function because the route exists and the role was not checked. Mass assignment means the client sends role=admin or price=0 and the server copies the body onto the record. Excessive data exposure means the response carries fields the screen does not show: full addresses, tokens, internal flags. Fix the server. Do not ask the mobile app to hide the extra fields. Hiding is not deleting. For each route, name the object, the owner rule, the role, and the response fields. Test a second user. Test a field the client should not set. Chapter 4 gave these bugs family names. Here they live on purpose-built APIs.

Rate limits, quotas, and abuse. A limit protects the dependency and the person. Login attempts, OTP checks, search, and bulk export need tighter budgets than a health check. Quotas can be per token, per IP, per merchant, per day. Return a clear slow-down, not a stack trace. Alert when one principal walks the whole id space. A Swiggy-shaped menu can be cached. A Paytm-shaped balance cannot be scraped by a loop that never trips a quota. Limits are not a substitute for authorization. A thief with a valid token and a polite rate still steals, only slower. Use both.

Schema validation and contract tests. Reject unknown fields if your threat is mass assignment. Enforce types, lengths, and enums. For GraphQL, cap depth and complexity so a small query cannot fan into a million joins. For gRPC, the protobuf contract is a start, not the authorization. Contract tests in CI fail the build when a response grows a new sensitive field or when a required auth header disappears. The contract is a security document. Review it like a policy.

Gateways and policy. An API gateway is a front door: TLS, authentication offload, rate limits, request size, allow-listed routes, and a place to kill a route without redeploying every service. It is not the only door. Services still check the token and the object, because a pod can be reached inside the cluster by something that skipped the gateway. Shadow paths, debug ports, and "temporary" admin routes are how the gateway's story becomes fiction. Generate routes from code or from the gateway config so the inventory is a file.

mTLS and workload identity. Between services, mutual TLS means both sides present certificates. The caller is not "the network." The caller is a named workload. Chapter 10 taught certificates. Chapter 15 taught the mesh idea in the cluster. Chapter 18 taught the cloud role. Here they meet: service A may call service B's getOrder and may not call refund. Mesh authorization is that sentence enforced beside the certificate. Certificate lifetime is short. The identity in the certificate matches the service account, not a shared cluster wildcard you copied from a sample. A plaintext HTTP call inside the cluster is a hallway without names. Prefer the named hallway even when the network policy already narrows the path.

Replay, nonce, and idempotency. A signed request captured on the wire can be sent again. A nonce, a timestamp, and a short validity window make the copy stale. An idempotency key makes a retried payment create one charge, not two. Store the key until the business action is finished. BHIM UPI-shaped and PayFast-shaped flows need this more than a read of a menu. Do not invent a new signature scheme. Use the standard your provider documents, check the timestamp with a tight window, and store keys in a place the handler can see. Log duplicates as duplicates, not as second successes.

Webhook signing. A webhook is an API you did not call. The other side calls you. Verify a signature with a secret you rotate, check the timestamp, and reject unsigned bodies. Zomato-shaped or Swiggy-shaped partner events should not update an order because the JSON "looks right." Respond quickly and do the work on a queue so the sender retries safely. Idempotency applies here too. A webhook endpoint is internet-facing. Inventory it. Give it an owner. Do not leave the signing secret in the repository.

GraphQL depth and cost. One endpoint can still be a hundred queries. Limit depth. Limit the cost of fields that hit the database. Disable introspection on public production if your policy says so, knowing attackers can guess fields anyway. Authorization is per field when fields have different owners. A public profile and a private address are not the same resolver. Timeouts and pagination are kindness and defense together.

Inventory, shadow, and zombie APIs. Shadow means deployed and not in the catalog. Zombie means deprecated and still answering. Both skip the gateway rules you were proud of. Discover routes from code, from the gateway, from load balancer logs, and from the cluster ingress. Diff them. Anything in the logs and not in the catalog is a ticket. Anything in the catalog and not called for a quarter is a candidate to turn off. Amazon-shaped estates and Infosys-shaped client handovers fail here when the diagram is a year older than the cluster. Assign an owner to every route. An ownerless API is a future incident with no phone tree, and you still will not put a phone number in the runbook's public copy. Put a team name.

Service authorization, said twice. The gateway authenticates. The service authorizes the object. The mesh authorizes the hop. If you only do the first, a pod that can route inside the cluster skips your front door. If you only do the third, a stolen user token still walks order ids. Write the three checks in the route's notes so a reviewer can see which one a pull request forgot. A Paytm-shaped refund route fails closed when any one of the three is missing. A Swiggy-shaped read route may be cached at the gateway and still must not be cached across users. Shared caches that ignore the subject are a quiet BOLA. Key the cache on the caller, or do not cache the private response.

OWASP API Security Top 10, as names. Learn the list your year uses, and expect these ideas: broken object authorization, broken authentication, broken object property authorization, unrestricted resource consumption, broken function authorization, unrestricted access to sensitive business flows, server side request forgery, security misconfiguration, improper inventory, and unsafe consumption of APIs. Names help you talk. They do not replace the object check. When a scanner shouts a name, reproduce it in your lab app with two users and a fix, not against a company on the internet.

A tiny object check. Lab only.

caller = verified subject from the token
order = load(order_id)
if order.owner != caller and "support-read" not in caller.roles:
    deny and log
return only fields marked public for this role

Support-read is a role from Chapter 19, just-in-time, not a flag in the request body. Log the deny with the subject and the order id, never with the token itself. A reviewer should see the denial in the same pull request that adds the route.

20.3 How.

1. Write "own lab only" and list every route your lab app actually serves.
2. Mark each route public, user, or service-to-service.
3. Put a gateway or a single front handler in front of the user routes.
4. Terminate TLS at that door and do not leave a second plain port publishing the same routes.
5. Issue lab tokens from your Chapter 19 identity provider, not from a hard-coded secret in the app.
6. Validate issuer, audience, expiry, and a pinned algorithm on every call.
7. Reject a token you edit by hand in the lab. Confirm the denial is logged.
8. Add an owner check on one object route. Prove user A cannot read user B's object.
9. Add a function check. Prove a normal user cannot call an admin route.
10. Strip or reject fields the client is not allowed to set, such as role or price.
11. Return only the fields that role needs. Look at the JSON, not only the screen.
12. Add a rate limit on login or on the object route and prove the limit trips.
13. Add schema validation for type and length on one input.
14. If you have GraphQL in the lab, set a depth limit and a timeout.
15. Issue a lab certificate pair and call one internal route with mutual TLS, or document the mesh policy you would apply.
16. Authorize that internal call by service name, not by "any pod in the namespace."
17. Add an idempotency key to one lab write and show that a retry does not double the write.
18. Verify a lab webhook signature and reject an unsigned body.
19. Diff gateway routes against application routes. File a ticket for any shadow route, even a health debug route.
20. Turn off one zombie route and prove clients that should have moved are updated.
21. Put a contract test in CI that fails if the auth header becomes optional.
22. Log a correlation id from the gateway into the service.
23. Review tokens in logs and remove any log line that prints the full token.
24. Write the owner rule for each sensitive route in the same file as the route list.
25. Stop. No scanning of real APIs, no guessing of order ids on systems you do not own, no copied partner webhooks aimed at production.

20.4 A real public incident.

Name. T-Mobile API data incident disclosed in January 2023.

Date. T-Mobile said it identified the activity on 5 January 2023, and that the actor first retrieved data through the API starting on or around 25 November 2022. T-Mobile filed a public current report with the U.S. Securities and Exchange Commission on 19 January 2023.

Source. T-Mobile US, Inc. Form 8-K filed 19 January 2023 (SEC accession 0001193125-23-010949). This chapter uses that filing. It does not describe requests, parameters, or how the API was called.

What happened. T-Mobile said a bad actor was obtaining data through a single API without authorization. The company said that within a day of learning of the activity it traced the source and stopped it, and that the investigation then indicated the activity was contained, with no evidence at that time that systems or the network were breached beyond that API. T-Mobile said the API did not provide payment card data, government identifiers, passwords, or other financial account information. It did provide a limited account set: name, billing address, email, phone number, date of birth, account number, and information such as line counts and plan features. T-Mobile said data was obtained for approximately 37 million current postpaid and prepaid accounts, though many accounts did not include the full set. An API that can answer for tens of millions of people is not a small side door. Authorization, inventory, and rate or anomaly detection on that one route were the controls that had to exist before the filing.

How to stop.

1. Disable or block the abused route and revoke credentials tied to it.
2. Confirm whether other routes share the same missing check.
3. Preserve gateway and application logs for the window you know and the window you fear.
4. Notify customers and regulators as the law requires. Do not invent a second channel that asks people for passwords.
5. Rate-limit and authorize the replacement route before you turn it back on.

How to prevent it again.

1. Object-level and function-level checks on every route, tested with two principals.
2. An inventory that includes this API, an owner, and an expected caller.
3. Tokens with audience and expiry, or an equivalent authenticated service identity. A route that answers personal data to an unauthenticated caller is a defect, not a convenience.
4. Quotas and alerts when one caller reads far beyond a normal session.
5. Response fields limited to the purpose. Do not return a full profile because a screen needed a name.
6. Practice the two-user test in your lab. Do not call real telecom or payment APIs to see what they return.

This is for learning only, and only inside a system you are allowed to test.

20.5 A simple picture.

The caller presents a token. The gateway checks the front door. The service checks the object. The mesh names the workload. A route with no owner never reaches that stack because the inventory removed it.

20.6 A short animation.

A token for the wrong audience fades and never opens the order. The right audience still has to pass the owner check.

20.7 Conclusion.

Raju's second lab user got a denial, and the denial was the point. Guru-ji closed the route list. "The streets have names now. In Chapter 21 we watch the rooms while they run. A perfect token does not stop a workload that starts a program the image never contained."

• A token proves the caller only after issuer, audience, expiry, and algorithm are checked.
• BOLA and BFLA are object and function checks. Mass assignment is a field check.
• Gateways help. Services and meshes still authorize.
• Idempotency, webhook signatures, and inventories close the side streets.
• T-Mobile's 19 January 2023 filing is the single-API lesson, about 37 million accounts.
• Own lab only.

END OF CHAPTER 20 MASTER CLASS