Chapter 10: Secrets, Keys, Certificates and PKI

10.1 Why.

Welcome to Chapter 10.

I am Ravindra Bagale.

Raju needed a database password for the lab API. He had no engineering degree. He pasted the password into a WhatsApp chat with himself so he would not forget it. Then he pasted it into a ticket. Then he committed a .env file "just for today." Guru-ji held up a hand.

"A secret that travels by chat is a postcard," Guru-ji said. "A private key that sits in Git is a public key with a disguise. Paytm cannot keep wallet trust that way. Amazon cannot keep customer trust that way. BHIM UPI cannot move money on a certificate nobody can revoke cleanly. Swiggy, Zomato, Netflix, Infosys, and PayFast all drown if the signing key for production is a screenshot."

Raju deleted the chat message. The password was still in the ticket and in Git history. Chapter 6 had already taught revoke first. Today Guru-ji wanted the larger map: what kinds of secrets exist, where they live, how certificates chain to trust, and how rotation happens without an outage story that lights up the Magical Code Library in red.

spring.datasource.password=${DB_PASSWORD}

App starts; no password visible in application.properties

import os
DB_PASSWORD = os.environ['DB_PASSWORD']

KeyError: DB_PASSWORD   (if you forgot to set it, which is good: it fails loudly)

const pass = process.env.DB_PASSWORD;

undefined   (if not set)

dotnet user-secrets set "Db:Password" "..."

Successfully saved Db:Password = ... to the secret store.

pass := os.Getenv("DB_PASSWORD")
if pass == "" { log.Fatal("DB_PASSWORD not set") }

DB_PASSWORD not set

DB_PASSWORD=${DB_PASSWORD}

php artisan config:cache   Configuration cached successfully.

bin/rails credentials:edit

config/credentials.yml.enc is encrypted; config/master.key is git-ignored

aws secretsmanager get-secret-value --secret-id prod/db

"SecretString": "{...}"  (only for a role with permission)

10.2 What.

Secrets taxonomy. Passwords authenticate people and sometimes services. API tokens authorize programmatic calls. Symmetric keys encrypt or MAC data. Asymmetric key pairs sign and decrypt with separate halves. Certificates bind a public key to a name and an issuer. Session tokens and refresh tokens are short-lived cousins. One-time passwords are proof of possession for a moment. Treat each class with a lifetime and a storage rule. A password in a human password manager is fine. A password in a mobile screenshot shared to a group is not.

Secret managers. HashiCorp Vault and cloud-native secret services exist so apps fetch secrets at runtime instead of baking them into images. The app authenticates to the manager with an identity: an instance role, a Kubernetes service account, or a CI OIDC token. The manager returns a secret over TLS. Access is audited. Revocation is a policy change, not a scavenger hunt across laptops. In a lab, run a manager or use a cloud secrets service on an account you own. Do not load real production keys into a learning vault.

Dynamic secrets and short-lived credentials. Some managers can mint a database user that expires in an hour. Cloud roles assumed through OIDC last minutes. Short life shrinks the window when a leak matters. Long-lived static keys are easier to paste and harder to forget. Prefer short. When you must use static, rotate on a calendar and on every suspected leak.

Encryption at rest and in transit; envelope encryption. In transit means TLS on the wire, as Chapter 9 required. At rest means disks, object stores, and database files encrypt so a stolen disk is not a stolen clear archive. Envelope encryption means a data key encrypts the bytes, and a master key in a KMS wraps the data key. You rotate the master carefully. You rewrap data keys without always re-encrypting every byte on day one. Understand the hierarchy before you invent a homemade crypto protocol. Homemade crypto is a classic own-goal.

KMS and HSM concepts. A Key Management Service stores and uses master keys behind an API. A Hardware Security Module keeps key material in tamper-resistant hardware. Higher assurance costs more and slows some operations. Payment and certificate-authority private keys often demand HSM-backed protection. App data keys may live in cloud KMS with good IAM. Least privilege on who may encrypt, decrypt, and administer is the real control. A KMS key that every role can decrypt is a decorative lock.

PKI, certificate authorities, and trust chains. Public Key Infrastructure is how names and public keys earn trust. A certificate authority signs certificates. A root CA is trusted by browsers and systems, often via a trust store. Intermediate CAs sit under roots so roots can stay offline. A leaf certificate names your service. Clients verify the chain: leaf to intermediate to a trusted root, plus time validity, revocation status when checked, and the name matching the host they called. If any link lies, TLS becomes costume jewelry.

Issuance, renewal, revocation, rotation. Issue certificates with automation where you can: ACME for public names, or an internal CA for private mesh names. Renew before expiry. Monitor expiry; an expired certificate is an outage that looks like a security event. Revoke when a private key leaks or a name is retired. Rotation means replacing keys and certificates on a schedule even without a known leak. Practice rotation in the lab until it is boring.

mTLS and service identity. Mutual TLS means both sides present certificates. In a service mesh, identities map to certificates. A caller without the right identity fails. Combine with authorization: the inventory service may call read on catalog, not refund on payment. Certificate identity without object checks repeats Chapter 3's mistake in a tuxedo.

Secret scanning. Scan source, images, and logs for shapes of keys and tokens. Push protection blocks a commit that looks like a cloud key. Scanning is a seatbelt. It will miss novel formats. Design so secrets never enter the repo. Chapter 6 already banned .env in Git. Here you also ban keys in container layers, in CI logs, and in exception messages.

Kubernetes secrets limitations. A Kubernetes Secret object is base64-encoded by default, not encrypted with a personal puzzle. Anyone who can read secrets in the namespace can read the bytes. Enable encryption at rest for the API store. Prefer mounting secrets from an external manager through CSI or injecting at runtime. Restrict RBAC so a microservice account cannot list all secrets in the cluster. Do not paste production keys into a manifest in a ticket.

Injection patterns. Environment variables are simple and easy to leak into logs and child processes. Files mounted with tight modes are better when the runtime supports them. Sidecars and CSI drivers can refresh secrets. Templates that render secrets into config files at start must not leave world-readable copies. Prefer libraries that read from the manager directly.

Rotation without downtime. Overlap old and new. Accept both passwords during a window, or use dual certificates, then drop the old. For tokens, cut the old token only after every consumer fetched the new. Practice on a lab database user. Document the steps. The first production rotation should not be improvisation at 2 a.m.

Backup and disaster for key material. If you lose the only copy of a CA private key, you may have to re-issue a universe of certificates. If someone else gains that copy, you must revoke a universe. Back up CA material with dual control and encryption. Store recovery procedures offline enough to survive the same fire that takes the primary. Test restore on a lab CA, not by hoping.

Teaching sample: fetch a secret in a lab app. This is a sketch for a machine you own.

# lab only — fake names, no real cloud account required to read this lesson
import os

def db_password():
    # Prefer a manager SDK in real code. Here we show the rule:
    # the password is not a string literal in source.
    value = os.environ.get("LAB_DB_PASSWORD")
    if not value:
        raise RuntimeError("LAB_DB_PASSWORD missing; inject from your secret manager")
    return value

The point is the absence of a literal. Wire the environment from a manager in your lab compose file. Do not commit the compose override that holds the real value.

DigiNotar-shaped fear, in one sentence. If a public CA is compromised, browsers must distrust it, and every site that depended on it must replace certificates. That is why root keys live in HSMs and ceremonies, and why intermediates exist.

Paytm and BHIM UPI stakes. Payment intents and wallet balances are assets from Chapter 3. The keys that authorize a credit to a merchant account are higher still. A leaked merchant API token can drain trust even when the TLS certificate is perfect. Separate duties: the person who can mint a production token should not be the only person who can deploy, and the deploy role should not be able to export the CA key. Amazon-shaped retail and Netflix-shaped streaming use the same separation under different product names.

Infosys-shaped client boundaries. On a project serving two banks, secrets for bank A must not be readable by the namespace for bank B. That is multi-tenant secrets hygiene. Name the secret paths with tenant ids. Audit list and read operations. A shared "ops" vault folder for all clients is convenience and a breach waiting for a calendar date.

Logging discipline. Frameworks love to print configuration on startup. Configure them so secret-bearing fields redact. A single debug flag in production has published tokens into centralized logs that every engineer can query. Central logs need access control too. Treat log stores as secret-adjacent.

Human processes. Break-glass accounts exist for outages. They must be rare, monitored, and rotated after use. Dual control for CA ceremonies means two people. A ceremony checklist is not bureaucracy. It is how you avoid DigiNotar-shaped regret.

What this chapter refuses to teach. How to forge certificates, how to extract keys from HSMs, how to bypass TLS for third-party sites, and how to phish vault credentials. Those topics are outside the Magical Code Library on purpose.

Certificate transparency and public logs. Public CAs often log issued certificates to transparency logs so domain owners can notice surprise certificates for their names. Monitor names you own. A surprise certificate can mean mis-issuance or an attacker using a lookalike process. Internal CAs may not use public CT; then your own inventory of issued leaves must be strong.

Machine identity versus human identity. Humans use SSO and MFA. Machines use roles, instance identity documents, workload identity federation, and certificates. Do not share a human's refresh token to run a batch job. Mint a machine identity with the smallest scope. PayFast-shaped merchants integrating webhooks verify signatures with keys that are machine secrets, not a founder's laptop password.

Envelope encryption walk-through for a lab object. Generate a data key. Encrypt the file with the data key. Encrypt the data key with KMS. Store ciphertext plus wrapped key. To read, decrypt the data key with KMS, then decrypt the file. Delete clear data keys from memory when the library supports it. Never log the data key. This pattern appears in cloud object encryption and in many application-level designs.

SSH certificates as a cousin of PKI. Some estates issue short-lived SSH certificates instead of scattering permanent authorized_keys files. The same morals apply: protect the SSH CA key, keep lifetimes short, and revoke by stopping issuance and denying serials when your stack supports it. Chapter 8 hardened SSH; this chapter explains why a long-lived user key in a hundred images is a secret sprawl problem.

Secret sprawl metrics. Count how many places a production DB password still lives: vault, an old CI variable, a helm value in a private repo, a runbook screenshot, an engineer's password manager shared vault. The goal is one authoritative place plus audited checkout. Sprawl is the enemy. Swiggy-shaped growth and Zomato-shaped growth both tempt sprawl when teams move fast. Put a secrets review on the architecture checklist beside the threat model.

Disaster story you want to avoid. The only person who knew the CA passphrase left the company. The HSM backup was never tested. A leaf expired on a festival weekend. That is not bad luck. That is missing rotation, missing dual control, and missing restore drills. Write the drill date in the calendar when you create the CA.

10.3 How.

1. Inventory secret types in your lab app: DB password, API token, TLS key, and signing key.
2. Write a lifetime beside each: hours, days, or ninety days.
3. Stand up a secret manager or cloud secrets service on an account you own.
4. Authenticate the lab app with a role or service account, not with a long-lived personal key pasted into code.
5. Remove any secret literals from source and from container build args.
6. Turn on secret scanning and push protection on the lab repository.
7. Configure TLS on the lab HTTPS endpoint with a certificate whose private key lives in the manager or in a restricted file mode.
8. Document the chain: leaf, intermediate, root.
9. Automate renewal reminders or ACME renew for names you control.
10. Practice revocation: replace a lab key, reload the service, confirm the old key no longer works.
11. Enable KMS-backed encryption for a lab disk or bucket and prove a role without decrypt permission cannot read clear data through the API path you test.
12. Draw envelope encryption: data key wraps bytes; master key wraps data key.
13. For a lab mesh or two services, enable mTLS and verify a third service without a cert cannot join.
14. Restrict Kubernetes RBAC in a kind/minikube lab so only one service account reads its secret.
15. Prefer file inject or SDK fetch over echoing secrets into build logs.
16. Mask CI variables in the pipeline UI and confirm a debug echo does not print them.
17. Write a dual-run rotation playbook for the DB password and rehearse it once.
18. Back up lab CA material encrypted to a second location you control; test restore.
19. Delete secrets from tickets and chat; if they appeared, rotate.
20. Add CODEOWNERS for any path that could hold material templates.
21. Educate the team in one page: how to request a secret, how never to paste one.
22. Review IAM on the KMS key monthly in the lab until the habit sticks.

23. Stop. Do not attempt to obtain, crack, or misuse certificates or keys from Amazon, Paytm, or any foreign system.

24. Enable certificate-expiry monitoring for every lab name you serve.

25. List all CI variables in the lab project and delete unused ones.
26. Replace one static cloud access key with an OIDC role assumption if your lab cloud supports it.
27. Confirm backup of vault or KMS recovery procedure is readable by a second person.
28. Redact secrets in a sample exception message by coding a safe logger in the lab app.
29. Review who can get secrets in the lab Kubernetes namespace and cut the list.
30. Stop. No key material from foreign companies enters your notes.

aws secretsmanager rotate-secret --secret-id prod/orders-db

{"ARN": "...", "Name": "prod/orders-db", "VersionId": "..."}

Database user "report-ro" expires in 6 hours

Auth token update: ROTATE strategy applied

sudo certbot renew --dry-run

Congratulations, all simulated renewals succeeded

keytool -list -v -keystore app.p12 | grep Until

Valid from: ... until: Sun Jan 10 2027

kubectl get secret db -o yaml

data: password: cGFzc3dvcmQ=   (base64 is not encryption)

echo "${{ secrets.DB_PASSWORD }}"

***   (masked in the log)

10.4 A real public incident.

Name. DigiNotar certificate authority compromise.

Date. DigiNotar detected a compromise in July 2011. Public distrust and government disclosures unfolded through August and September 2011, including drastic browser and national responses that autumn.

Source. The public report prepared for the Dutch government by Fox-IT on the DigiNotar breach, contemporaneous advisories from browser vendors that distrust DigiNotar roots, and reporting by The Register and other outlets in September 2011. This chapter uses those public materials. It does not provide a method to compromise a CA.

What happened. Intruders breached DigiNotar's systems and caused fraudulent certificates to be issued for high-value names. Because DigiNotar was a trusted public CA, software that trusted its root could accept those certificates as valid. The failure was catastrophic for a PKI: the trust anchor itself was tainted. Browsers and platforms moved to distrust DigiNotar. The Netherlands faced emergency measures for certificates used in government-related services. A CA is not an ordinary web app. Its private keys are nation-scale secrets. Weak segregation, monitoring, and hardening around issuance turned into a global trust emergency.

How to stop.

1. Distrust the compromised roots in clients and enterprises.
2. Revoke known bad certificates and replace legitimate ones from a clean CA.
3. Take the compromised issuance infrastructure offline for forensic work.
4. Communicate clearly to relying parties what to distrust and what to install instead.

How to prevent it again.

1. Protect CA private keys in HSMs with dual control and audited ceremonies.
2. Separate issuance networks from general IT; monitor for unusual issuance.
3. Prefer short-lived certificates so revocation windows hurt less.
4. Maintain agility to distrust a CA and shift traffic.
5. For your own services, minimize dependence on a single poorly governed CA.
6. Study DigiNotar as history. Do not test CA defenses on systems you do not own.

This is for learning only, and only inside a system you are allowed to test.

ls -l /etc/letsencrypt/live/example.in/privkey.pem

lrwxrwxrwx ... -> ../../archive/.../privkey1.pem  (target file mode 600)

server.xml: certificateKeystorePassword should come from a protected file, not Git

JWT header: {"alg":"RS256","kid":"2026-10"}

10.5 A simple picture.

Root signs intermediate. Intermediate signs the service. The private key stays in the secret manager. Git and chat are marked as forbidden landing zones.

10.6 A short animation.

A private key tries to visit a chat. The stamp says no. The vault is the correct address.

10.7 Conclusion.

Raju rotated the lab password and put the new value only in the manager. Guru-ji checked Git one more time and found no literals. "Keys are oxygen," Guru-ji said. "Invisible when present, fatal when dirty. In Chapter 11 we put these secrets into CI/CD without letting the pipeline become a second company."

• Classify secrets and give each a lifetime and a home.
• Prefer secret managers and short-lived credentials over static pasted keys.
• PKI chains trust from root to leaf; protect CA keys like crown jewels.
• Rotate, revoke, scan, and never store private keys in Git or chat.
• DigiNotar, 2011, is the compromised public CA lesson.
• Short lifetimes and dual control beat heroic recovery after a leak.
• Own lab only. Practice revoke and renew until both are boring.

END OF CHAPTER 10 MASTER CLASS