Chapter 27: SRE and Security Together

27.1 Why.

Welcome to Chapter 27.

I am Ravindra Bagale.

Raju had no engineering degree. He had a security change he wanted to ship at the worst minute of the day, and a reliability page that said the error budget was already gone. He was about to turn the control off so the graph would go green. Guru-ji put a hand on the laptop lid.

"Reliability without security is a city with the gates open and the trains on time. Security without reliability is a gate that crushes the people it was meant to protect. You share one budget. You spend it on purpose."

Raju asked who wins the argument. Guru-ji said nobody wins by shouting. "Suppose we are Netflix and a control adds so much latency that the evening stream fails. Suppose we are Swiggy and a lunch-hour deploy of a new rule blocks every order. Suppose we are Paytm or PayFast and a failover region comes up with last year's admin role because the security review only looked at the primary. Suppose we are Amazon and a capacity guess ignores a flood of junk traffic. Suppose we are Zomato and the backup restores, but the restored database trusts the whole office network. Suppose we are Infosys and the client SLO is in the contract, and the security exception is a verbal promise. Suppose we are BHIM UPI or WhatsApp-shaped systems and a rate limit is missing, so one bad client becomes everyone's outage. Write the trade. Then choose."

27.2 What.

SLI, SLO, and SLA. A service level indicator is a measurement: the share of requests that succeed, the share that finish under a latency line, the share of minutes a control was actually enforcing. A service level objective is the target you set for that indicator over a window, such as a month. An agreement is what you promised a customer, which may be stricter or looser than the internal objective. Security joins this vocabulary. "Share of admin logins with a second factor" can be an indicator. "Share of deploys that passed the signed-image check" can be an indicator. If you only measure uptime, you will optimize uptime and call a wide-open admin panel a success.

Error budgets. The error budget is the amount of unreliability you can spend before you stop and fix. Feature work spends it. So does a risky security rollout. So does a patch reboot from Chapter 25. When the budget is gone, the paved response is to halt risky changes, including the ones with a security logo, unless the change is itself what stops the bleeding. A freeze that blocks an emergency patch for a known-exploited internet-facing bug is a bad freeze. Write that exception in advance. A team that spends the whole budget on features and then refuses a patch is also failing the objective. Security and reliability read the same number.

Trade-offs you must name. A web application firewall rule can block attacks and can also block customers. A strict mutex on deploys can slow a hotfix. Extra encryption can cost CPU. A short session can annoy a kitchen operator at Zomato and can also shrink the window a stolen cookie is useful. Name the user harm and the attacker harm in one paragraph before you ship. If you cannot name both, you are not ready. Prefer a canary. Prefer a rollback that is one flag, not a rebuild you have never practiced. Netflix-shaped progressive delivery exists so a bad rule dies small.

Chaos, done safely. Chaos engineering injects failure on purpose so you learn. In this course the purpose is your lab. Kill a lab dependency. Fill a lab disk. Expire a lab certificate. Refuse a lab token. Watch whether the service degrades or falls over, and whether your alerts tell the truth. Do not inject failure into a network you do not own. Do not call an unsolicited load test an experiment. Security chaos has the same fence. A test that a revoked certificate is rejected belongs in your lab. A test that tries to break someone else's login does not.

Dependency failure and graceful degradation. Every real service depends on identity, a database, a queue, and often a vendor. Decide what still works when each one is sick. Maybe you can show a cached menu and refuse new payments. Maybe you must stop entirely because a wrong payment is worse than no payment. Paytm-shaped and PayFast-shaped flows usually fail closed. A Swiggy-shaped status page might fail open on a read and fail closed on a write. Document the choice. Rate limits and quotas are part of degradation. They keep one client, one bug, or one retry storm from eating the budget. Set them in the lab and watch a friendly test client get a calm error instead of taking the process down.

Backups, recovery objectives, and failover. A recovery point is how much data you can bear to lose. A recovery time is how long you can bear to be down. Security sits inside both. Backups must be immutable enough that a destructive program cannot delete them with the same credential that deleted production. Restore tests must check that the restored system does not come back with a public bucket, a shared admin password, or a disabled log. Multi-region failover is a security design, not only a map. The second region needs the same identity baseline, the same private networking, and the same patch level. A region you have never failed over to is a region that will improvise, and improvisation invents doors.

Secure incident management. Chapter 23 gave you the response loop. SRE adds the operational spine: roles, a status channel, a decision log, and a rule for when you communicate outside the team. Security incidents and reliability incidents often arrive as the same page. A spike might be a bug or a flood or a credential being abused. The first minutes are the same: protect people, protect evidence, stop the bleeding with the least new damage. Do not wipe logs to make the graph pretty. Do not share secrets in the incident channel. Afterward, write one lesson that becomes a test or a template change. A lesson that lives only in a slide will visit you again.

Production readiness. Before a lab service is called production, even in a classroom sense, walk a list. Owner. SLO. Alert that a human reads. Dashboards that do not include secrets. Admin path with a second factor. Backup tested. Dependencies named. Rate limit on. Security review of the data class. Rollback practiced. On-call knows how to disable a bad rule. Infosys-shaped handovers fail when this list is tribal knowledge. Put it in the catalog from Chapter 26.

Capacity and abusive traffic. Capacity planning asks what happens when legitimate traffic doubles. Resilience against junk traffic asks what happens when illegitimate traffic doubles. You do not need to build a flood against yourself beyond a modest lab client. You do need quotas, autoscaling with a ceiling, and a way to shed load. A ceiling matters. Unlimited scale can turn a flood into an unlimited bill and an unlimited outage. DDoS resilience is a design topic: absorb or shed at the edge, protect origin, keep identity available so operators can still get in. It is not an invitation to generate a flood toward any company, including Amazon, Netflix, or anyone else.

Toil. Toil is manual work that scales with tickets: copying logs by hand, rotating a key by pasting, clicking the same approval because the tool cannot. Toil hides security work until it is skipped. Automate the paved parts. Keep humans on the judgments: accepting a risk, calling an incident, talking to a customer. A runbook that is twenty screenshots will not be followed at dinner time. A runbook that is one command with a logged reason might be.

The shared calendar. Patch Tuesday, template releases, failover drills, and feature freezes belong on one calendar. A security team that surprises SRE at 6 p.m. and an SRE team that freezes every patch both create incidents. BHIM UPI-shaped payment windows and WhatsApp-shaped peak messaging hours are constraints you design for. Canary before the peak. Finish or roll back before the peak. Write that into the production review.

A teaching sample: one objective card. Lab only.

service: lab-orders
sli: share of requests served in under 300ms, and share of deploys with a signed image
slo: 99 percent on both, over 30 days
budget rule: no risky change when either budget is gone, except a known-exploited patch on an exposed lab service
rollback: one flag returns the previous digest
failover: second lab VM, same identity baseline, synthetic data only

Seminar notes. Guru-ji made Raju write the user sentence before the control sentence. "A customer can place a synthetic order." Then the security sentence. "A second lab user cannot read that order." Both sentences need an indicator. If you only alert when the order fails, you will ship a control that makes orders fail and call the silence of attackers a victory. If you only alert when a scan is red, you will not notice that the city is down. Put the two sentences on the production-readiness card and refuse to call the service ready when either sentence lacks a measurement, an owner, and a rollback.

Chaos in this course stays boring on purpose. Expire a certificate. Revoke a lab token. Fill a disk. Delay a dependency until the timeout fires. Each drill answers a design question. Did we fail closed on the payment-shaped write. Did the operator still have a path to the kill switch that does not ride the broken dependency. Did the log survive. A drill that you cannot explain to a classmate without whispering a trick is the wrong drill. Write the result as a change to the template or the runbook. Toil is the enemy of the second drill. If rotating the lab certificate takes an afternoon of clicks, the emergency rotation will be skipped. Automate the rotation path once, log the reason, and keep a human on the decision to start it.

Capacity has a ceiling. Autoscaling without a ceiling turns a retry storm or a flood of junk into an unlimited bill and a second outage. In the lab, set a small ceiling and prove the calm error appears. Rate limits are part of security and part of reliability because they are the same knob. A limit that is per user protects neighbors. A limit that is global can lock out everyone when one client misbehaves, so design the key of the limit with the same care you design a primary key. DDoS resilience, as a concept, means the edge can shed load, the origin is not exposed just because the edge is, and operators can still sign in. It does not mean you generate traffic at a company. Netflix-shaped evenings and Swiggy-shaped lunch hours are when you do not experiment. Canary earlier. Finish earlier.

Multi-region security is a checklist you can rehearse with two virtual machines. Same patch level. Same identity baseline. Same private networking. Different credentials. Logs still leave both. The failover runbook names who may invoke it and how you know the second region did not come up with a public store. If the second region is a snowflake, it is not a region. It is a hope with an IP address. Infosys-shaped contracts that promise a recovery time need this rehearsal on a calendar, not a paragraph in a proposal. When the error budget is healthy, spend a little of it on the drill. When the budget is gone, the drill waits, unless the drill is the only way you know you can recover. Write that rule down so two teams do not argue it during the page.

27.3 How.

1. Pick one lab service you already run and write "own lab only" in its notebook.
2. Choose one user-facing indicator, such as success rate or latency, and record how you measure it.
3. Choose one security indicator, such as signed deploys or second-factor admin logins, and record how you measure it.
4. Set a modest objective for each over a short window you can actually observe.
5. Compute a toy error budget on paper for the week.
6. Add an alert that fires when the user-facing indicator burns budget too fast for a lab.
7. Confirm the alert uses a channel you control and does not include secrets.
8. Write the rule for what kinds of changes stop when the budget is gone.
9. Write the exception for an emergency patch, and who may invoke it.
10. Ship a harmless bad config to one lab instance only, and watch the indicator move.
11. Roll back with the method you documented, and time how long it took.
12. Kill one lab dependency on purpose and record whether the service fails open or closed.
13. Decide whether that choice is acceptable for the data class, and change it if it is not.
14. Set a small rate limit and prove a test client receives a calm rejection over the limit.
15. Take a backup, then change a config, then restore, and check that logs and access rules came back as designed.
16. If you have a second lab machine, rehearse a failover and compare its identity and patch level to the first.
17. Add the production-readiness list to the catalog row for this service.
18. Run one tabletop: a latency page that might be a bug or a stolen token. Write the first three questions you will ask.
19. Convert one toil step you felt this session into a script or a template change.
20. Put patch windows and drill windows on one calendar page.
21. Show a second person the objective card and ask them to point at the rollback.
22. Stop. Do not load-test, fail over, or tune controls on Swiggy, Paytm, Amazon, BHIM UPI, WhatsApp, Zomato, Netflix, Infosys, PayFast, or any service you do not own.

27.4 A real public incident.

Name. Cloudflare outage on 2 July 2019, caused by a web application firewall rule.

Date. The outage began on 2 July 2019. Cloudflare published an initial account the same day and a detailed account on 12 July 2019.

Source. John Graham-Cumming, "Cloudflare outage caused by bad software deploy," Cloudflare Blog, 2 July 2019, and "Details of the Cloudflare outage on July 2, 2019," Cloudflare Blog, 12 July 2019. This chapter does not repeat the rule pattern that caused the CPU exhaustion.

What happened. Cloudflare deployed a new managed firewall rule as part of ordinary protection work. The rule consumed so much CPU that HTTP services across the network failed, and visitors saw gateway errors. The company later said this was not an attack. The same detailed account listed contributing failures a reliability engineer will recognize: a change that could go global without a staged rollout, tests that did not catch dangerous cost, a rollback that was slow, alerts that were late, and operators who struggled to reach their own tools because the edge they were fixing was the edge they used. A security control, shipped to help, became the incident. The stop was to disable the bad ruleset worldwide, then restore a known good configuration after review. The deeper fix was engineering: stage the rollout, test the cost, practice the rollback, and keep a path for operators that does not depend on the thing that is on fire.

How to stop.

1. Identify the last change and halt further rollout.
2. Use the prepared kill switch for that control if you have one.
3. Restore the last known good configuration and confirm user traffic returns.
4. Keep logs. Do not debug by pasting secrets into a chat.
5. Tell users what failed and that you are in a known state, then investigate the rule in a lab copy.

How to prevent it again.

1. Canary security rules the way you canary features.
2. Test rules for cost and for false blocks, not only for whether they match a sample.
3. Keep a fast, practiced rollback that does not require the broken edge.
4. Put security releases on the error budget, with an emergency lane that is still staged when minutes allow.
5. Make sure operators can reach status and kill switches on a path that is not the customer path.
6. Practice on systems you own. Do not throw traffic or rules at Cloudflare or anyone else.

This is for learning only, and only inside a system you are allowed to test.

27.5 A simple picture.

One budget. Two spenders. Features and security changes both canary, both roll back, and both leave the operator a side door that still works.

27.6 A short animation.

A rule spreads to a small slice, the indicator worsens, and the slice returns to the previous digest.

27.7 Conclusion.

Raju rolled the bad flag back before the lab graph stayed red. Guru-ji tapped the objective card. "You did not win an argument. You kept a promise. In Chapter 28 the promise is about data. A database, a bucket, and a backup are not furniture. They are the reason the city exists."

• Indicators make security and reliability visible in the same units.
• Error budgets can freeze features and must not freeze an emergency patch without a written rule.
• Fail closed on payments. Test failover as a security design.
• Cloudflare, 2 July 2019, is the day a firewall rule became the outage.
• Own lab only.

END OF CHAPTER 27 MASTER CLASS