Chapter 26: Platform Engineering and Secure Developer Platforms

26.1 Why.

Welcome to Chapter 26.

I am Ravindra Bagale.

Raju had no engineering degree. He had twelve repositories and twelve slightly different pipelines. One ran tests. One skipped them because a deadline had been loud. One stored a long-lived key in a variable named temp. He called this freedom. Guru-ji called it twelve ways to forget a control you already knew how to build.

"A platform is a product," Guru-ji said. "Its customer is the engineer. If the secure path is longer than the clever path, the clever path wins. Pave the road. Put the guardrails in the template. Then freedom is which feature you ship, not whether you remembered to turn off a public bucket."

Raju worried that a platform team would become a ticket window. Guru-ji shook his head. "Suppose we are Amazon and every team invents its own account baseline. Suppose we are Swiggy and a new kitchen service copies last year's pipeline because the new one is a wiki page. Suppose we are Paytm or PayFast and the payment template is optional. Suppose we are Netflix and a scorecard shames teams without giving them a paved fix. Suppose we are Infosys and the client asks how a new service inherits identity, logs, and review rules on day one. Suppose we are Zomato, BHIM UPI, or WhatsApp-shaped systems and the portal shows owners, not only diagrams. Self-service with guardrails is the point. A locked door with no road is just a delay."

26.2 What.

Internal developer platform. An internal developer platform is the set of tools, templates, and runbooks a company offers so a team can go from idea to a running service without reinventing identity, build, deploy, and observation. It is not a single vendor logo. It might be a catalog, a pipeline library, an infrastructure module, and a chat channel with a human. The security goal is simple. The default service is already in a private network, already logging, already built by a reviewed pipeline, already using a short-lived identity. The team adds the business logic. They do not add a new theory of access control each time.

Golden paths and templates. A golden path is a supported way to build a thing: an HTTP service, a worker, a mobile backend, a function. The template includes the secure defaults. A health check. A non-root runtime user. A place for configuration that is not a secret in Git. A pipeline that runs the scanners from Chapter 12 and fails closed on the rules you promoted from guideline to standard. A deployment identity that can touch only that service. Document the path in the template's first page, not in a separate PDF nobody opens. If the template is stale, engineers will fork it and you will be back to twelve pipelines. Own the template the way you own a product. Version it. Deprecate old versions with a date.

Self-service with guardrails. Self-service means a developer can create a lab or a staging environment without waiting a week. Guardrails mean the creation API will refuse a public database, an open storage bucket, or an admin role. The refusal should explain the paved alternative in the same message. A hard no with no path teaches people to use a personal cloud account. That shadow platform will not have your logs. Policy-as-code from Chapter 16 belongs here, in the module the portal calls, not only in a scanner that comments after the mistake is live.

Secure-by-default pipelines. The pipeline template is where Chapter 11 becomes a habit. Protected branch. Second reviewer for the template itself. Dependency review. Secret scan. Signed artifact. Digest pin. No long-lived cloud key in variables. The application team should not be able to delete those steps without an exception that expires. They should be able to add tests. The difference matters. A platform that freezes every line will be bypassed. A platform that lets anyone comment out the secret scan will be theatre. Split the template into required stages and extension points.

Reusable policy and components. Write a logging sidecar, an identity helper, and a network policy once. Publish them as modules with a security review. Teams consume versions. When you fix a flaw in the helper, you bump the version and the paved road picks it up. A copy-pasted helper is a vulnerability backlog you created on purpose. Scorecards can show who is still on the old major version. The scorecard is a map for the platform team, not a wall of shame. Pair every red mark with a pull request the team can merge.

Catalogs, ownership, and portals. A catalog, in the sense of a Backstage-style portal or any serious service list, is the inventory Chapter 25 begged for. Each service has an owner, a tier, a repository, an on-call hint, and a link to its pipeline. Security feedback belongs in that portal: open SLA breaches, missing owner, exception about to expire. If the feedback lives only in a security inbox, the owner will see it late. Metadata is a control. A service with no owner is not done, and the platform should refuse to mark it production.

Ephemeral environments. A short-lived environment for a pull request is a gift and a risk. It must use fake data, not a copy of production customers. It must die on a timer. Its identity must not be the production role. Secrets for the ephemeral world are ephemeral too. A preview URL is still a URL. Do not leave it on the internet with admin credentials because the test was convenient. Swiggy-shaped and Zomato-shaped order flows can be exercised with synthetic orders. Paytm-shaped card flows use tokens from a test scheme, never real card numbers, and never a bucket of production logs.

Secrets and identity abstraction. The platform should hand the application a way to ask for a secret at runtime, and a workload identity that the cloud believes. The application should not know where the secret store lives, and it should not embed a key. Chapter 10 and Chapter 18 are the mechanism. The platform is how a new hire gets that mechanism without a week of folklore. Rotate the platform's own credentials on a schedule. The platform account is a crown jewel. Treat it as one.

Multi-tenancy. Many teams share a cluster, a CI fleet, or a registry. Isolation is the product. Separate namespaces or accounts, separate roles, and a default deny between tenants. A shared build cache must not let one repo poison another's artifact. A shared runner must not keep the previous job's secrets in the workspace. Platform SLOs include "one tenant cannot read another tenant's config." Test that SLO with two lab teams you control. Do not test it by poking at a neighbor you do not own.

Central controls and team autonomy. Central teams should hold the road surface: identity, policy modules, base images, registry, and the production deploy role. Product teams should hold the feature, the alert thresholds that match their users, and the decision to take a paved extension. Autonomy without a road is chaos. A road without autonomy is a queue. Write the split in a one-page contract. When a team needs an exception, use the same expiry rule as Chapter 24 and Chapter 25. Netflix-shaped speed comes from a good road, not from deleting the road.

Product thinking. Talk to developers. Measure time to first deploy, change failure on the template, and how often people bypass it. If bypass is common, the road is wrong. Fix the road. Do not write a longer policy. A security platform that only says no will be routed around. A security platform that makes the yes path safer than the side path will be used when nobody is watching. That is the goal.

Platform SLOs and governance. The platform itself has reliability and security objectives. The catalog is up. The pipeline queue is short. The base image is rebuilt when Chapter 25's emergency lane lights up. A change to the golden template is reviewed by someone who does not solely author it. Break-glass access to the platform is rare, logged, and expired. If the platform is down, say so, and say what manual path is still safe. A manual path that uses a personal admin key is not a fallback. It is an incident you scheduled.

What the portal must refuse. A create button that can mint an admin key is not self-service. It is a side door with a friendly label. List the refusals in the template tests. No public ingress on a data store. No static cloud key in the repo. No production identity on a preview. No deploy of an image that is not signed. No catalog entry without an owner and a tier. Each refusal has a sentence the developer can act on: which module to use, which field to fill, which lab role to request. Infosys-shaped delivery teams feel this on day one of a project. If the first hour is a scavenger hunt, the second hour is a personal script. Personal scripts do not get patches when the base image does.

Feedback that arrives where the work is. A scorecard in a monthly slide is a rumor. A failing check on the pull request is a decision. The platform should comment, in the repository, when the service drifts off the golden major version, when an exception is a week from expiry, and when Chapter 25's SLA is broken for that service. The comment should include the paved fix. Developers live in the pull request. Meet them there. Meet the on-call in the catalog. Meet an auditor in the evidence export from Chapter 24. Three doors, one set of facts.

The platform is software. It has dependencies, a threat model, and an on-call. A portal that can deploy to every tenant is a production system. Review its changes. Back it up. Limit who can edit the golden template. Log who generated a break-glass credential. Practice restoring the catalog. If the platform team is too small to be its own customer, run the golden path on the platform's own components first. A road the builders will not drive is a road they do not trust. Paytm-shaped payment templates and WhatsApp-shaped messaging templates can differ in data class and still share the same identity module. Sameness where it is safe, difference where the data demands it. That is design, not bureaucracy.

A teaching sample: a tiny golden service note. For a lab you own.

template: lab-http-service v3
defaults: private network, non-root, runtime identity, no static cloud key
pipeline: secret scan, dependency review, unit tests, signed image, digest deploy
owner: required field, deploy refused if empty
data: synthetic only in preview environments
exception: file with an expiry, not a comment that says skip

Seminar notes. Raju asked for a portal with every logo. Guru-ji asked for one paved create button that a tired person can use at the end of a long day. The button's success path is the security model. If success creates a private service, an owner, a pipeline, and a catalog row, you have a platform. If success creates an account and a wiki link, you have a brochure. Measure bypass for a month in the lab by counting services that did not come from the template. A bypass without a ticket is a defect in the road. A bypass with an expired exception is governance. Talk to the person who bypassed it. They are your most honest user. Fix the friction they hit, and keep the guardrail that made them bounce. That conversation is product management. It is also how Paytm-shaped payment templates stay stricter than a marketing-site template without becoming a different religion. Same identity module. Different data module. One catalog.

26.3 How.

1. Write "own lab only" at the top of a platform notebook.
2. List the steps a new lab service must take today, and mark which steps people skip when they are tired.
3. Choose one golden path: a small HTTP service.
4. Put a template repository under version control with a protected default branch.
5. Include a runtime user that is not root, and a health check.
6. Include a pipeline that runs a secret scan and fails the build on a test secret you plant on a branch.
7. Remove the planted secret and confirm the paved pipeline goes green only when it is gone.
8. Give the deploy step a lab role that can update one service and cannot administer the account.
9. Add an owner field and make the deploy script exit if the field is empty.
10. Document the template in the first page of its readme, including how to add a test and how not to delete the scan.
11. Version the template and deploy one sample service from it.
12. Create a second lab identity and prove it cannot read the first service's config.
13. Stand up a preview environment from a branch, load only synthetic records, and delete it at the end of the day.
14. Add a catalog row for the sample: name, owner, repo, pipeline URL.
15. Put one open finding from Chapter 25's style onto that catalog row so feedback sits where the owner looks.
16. Publish a shared module, even a tiny logging config, and consume it by version.
17. Bump the module and show the service picking up the bump through Git, not through a hand edit in production.
18. Write the split of duties: what the platform owns, what the app team owns.
19. Add an exception file format with an end date and a reviewer different from the author.
20. Measure time from empty directory to a running lab service, and write down the slow step.
21. Fix that slow step inside the template if the fix is a guardrail, not a bypass.
22. Review the platform role's permissions and delete any right the golden path does not need.
23. Stop. Do not deploy templates into Swiggy, Paytm, Amazon, BHIM UPI, WhatsApp, Zomato, Netflix, Infosys, PayFast, or any tenant you do not own.

26.4 A real public incident.

Name. Codecov Bash Uploader supply-chain compromise.

Date. Codecov said unauthorized changes to the Bash Uploader began on 31 January 2021. Codecov learned of the problem on 1 April 2021 and publicly notified customers on 15 April 2021.

Source. Codecov's "Bash Uploader Security Update," dated 15 April 2021, and Codecov's April 2021 post-mortem. A customer had compared a published checksum with the file they downloaded. This chapter does not repeat the altered script, any address it used, or any indicator list.

What happened. Codecov is a tool teams add to pipelines so coverage reports upload after tests. That is a paved path in miniature: one shared helper, used widely, trusted because it is the official way. Codecov reported that an error in its image-creation process let an outsider obtain a credential that could modify the uploader. The modified uploader could export environment variables from customers' continuous integration environments. Those variables often hold tokens and keys. The change sat in the shared path for weeks. Detection was not a grand monitor. It was a customer who checked a checksum. The lesson for platform engineering is uncomfortable and useful. The golden path is a high-value target. If everyone uses your template, a flaw in the template is everyone's flaw. Checksums, pinned versions, least privilege on the credential that publishes the template, and environments that do not hold production keys in the same variables as a coverage upload are platform controls. They are not paperwork.

How to stop.

1. Stop using the untrusted helper immediately and replace it with a version you have verified.
2. Rotate credentials that lived in the environments where the helper ran.
3. Review logs for use of those credentials after the window the vendor names.
4. Preserve evidence and follow the vendor's public notice. Do not turn the notice into an exploit lab.
5. Tell downstream owners which pipelines were affected.

How to prevent it again.

1. Pin and verify the helpers your golden path downloads. A checksum mismatch is a failed build.
2. Keep production secrets out of the environment of a reporting job that does not need them.
3. Protect the credential that publishes templates the way you protect a production deploy key.
4. Separate tenants so one pipeline cannot read another's variables.
5. Make the secure template easier than a copied script from a blog.
6. Practice in a lab you own. Do not probe Codecov or any company system you do not own.

This is for learning only, and only inside a system you are allowed to test.

26.5 A simple picture.

The side path is fast once and expensive forever. The paved road is the template, the identity, the scan, and the catalog owner.

26.6 A short animation.

A new service asks to exist. The template stamps the defaults. The catalog learns the owner.

26.7 Conclusion.

Raju deleted two of the twelve pipelines and pointed them at the template. The third still had a temp key. Guru-ji waited until Raju removed it without being asked. "That is product sense. In Chapter 27 we sit with the people who keep the road up at dinner time. Reliability and security share one error budget. A control that takes the city offline is not a win."

• Secure behavior should be the default of the template, not a wiki.
• Guardrails belong in the create API, with a paved alternative in the error.
• Catalog ownership is the inventory that patch queues need.
• The Codecov notice of 15 April 2021 shows a shared helper becoming everyone's risk.
• Checksums, small identities, and expiring exceptions keep the road honest.
• Own lab only.

END OF CHAPTER 26 MASTER CLASS