Chapter 23: Incident Response, Forensics, and Recovery

23.1 Why.

Welcome to Chapter 23.

I am Ravindra Bagale.

Raju had no engineering degree. He had a red alert from Chapter 22 and a finger already on the delete key. Guru-ji caught the wrist. "Deleting the pod deletes the witness. Response is a sequence: know who decides, keep the evidence, stop the bleeding, remove the cause, rebuild from what you trust, and write the lesson into a control. Panic is not a phase."

Raju looked at the lab pod that had started a shell. "I can kill it in one command."

"You can," Guru-ji said. "After you copy the alert, the image digest, and the cloud session id. Suppose we are Paytm and a payment key leaked an hour ago. Suppose we are Swiggy and the CI role that ships the dinner image can also rewrite itself. Suppose we are Amazon and an account trail goes dark. Suppose we are Netflix and the fastest fix is a rollback to an image that still contains the bad dependency. Suppose we are Zomato or WhatsApp-shaped systems and a bad deploy is easier to roll back than a compromised registry is to rebuild. Suppose we are BHIM UPI or PayFast and you rotate one secret but not the sessions that already used it. Suppose we are Infosys on a client clock and the client wants a timeline, not a hero story. The city survives the teams that rehearsed. It does not survive the teams that meet the sequence for the first time at midnight."

23.2 What.

The phases, in plain words. Preparation: roles, contacts, lab drills, logs that exist. Identification: is this real, what is in scope, what is the severity. Containment: stop the spread without destroying evidence you still need. Eradication: remove the access and the malware-shaped changes, rotate the secrets, close the hole. Recovery: return service from a trusted build, watch it, and only then call it done. Lessons learned: one control changes because of this incident, with an owner and a date. People abbreviate the list. Do not skip preparation and do not skip the lesson. A thrilling containment with no new control is a trailer for the next incident.

Who decides. Name an incident lead, a communications owner, and a technical lead before you need them. The lead can pull a service offline. The communicator talks to customers and to leadership with facts that have been checked. Engineers do not freelance status posts. Legal and privacy join when personal data is in play. A small lab still writes the three names. Infosys-shaped client work adds the client's incident owner to that page so you do not guess at midnight.

Severity. A shell in a sandbox with no data is not the same as a shell in a pod whose role can read payments. Use the join from Chapter 21. Severity drives the clock and the people, not the volume of the siren. Write three levels only. You can split them later. Level one: active access to customer or payment data, or a pipeline that can ship to production. Level two: a contained lab-like blast with a real secret exposed. Level three: a policy miss with no evidence of use. Do not open at level three because you hope.

CI and CD compromise. If the pipeline is the suspect, stop production deploys. Freeze the protected branch. Rotate CI credentials and the cloud role the pipeline assumes. Review recent merges and the identities that approved them. Rebuild artifacts from a known commit and a clean runner, not from the warm cache of the suspect job. Chapter 11 separated untrusted pull requests from release. If that separation failed, the recovery is to restore the separation and to treat every artifact from the window as untrusted. A green pipeline is not a character reference.

Secret leakage. When a key, a token, or a password reaches a commit, a log, a ticket, or a chat, it is burned. Revoke it. Rotate what it could open. Search logs for use. Search history, not only the tip of the branch. A private repository is still a leak. Chapter 6 and Chapter 10 already said this. Response adds a clock and a checklist so the human under stress does not stop after the first rotation and leave the session alive.

Cloud account compromise. Disable new keys. Revoke sessions. Look for new users, new trusts, trails turned off, and regions you do not use. Containment may mean a guardrail at the organization, not a plea to the account. Preserve cloud audit before you delete the strange user, or export the event first and then delete. Recovery is a rebuilt account path from infrastructure code you trust, plus a human review of every trust added during the window. Do not "click around until it looks familiar."

Kubernetes response. Capture the pod describe, the image digest, and the runtime alert. If policy allows, cordon and isolate before you delete. Network quarantine often beats instant deletion. Look for new role bindings, new service accounts, and workloads in namespaces you did not create. Recovery is a redeploy from the signed digest, onto nodes you believe, with the Chapter 18 role narrowed if the old role was the prize. A cluster you "clean" by hand without a rebuild will keep a souvenir.

Supply chain and artifacts. If the registry or the build is suspect, stop pulling the tag that floats. Pin the last digest you can justify. Compare signatures and SBOMs from Chapter 13. Notify downstream teams so they do not keep deploying the bad digest under a friendly tag. Recovery is a rebuild from source you diffed, on runners you reset. Rollback to a previous tag is right when the previous digest is clean and wrong when the registry itself was the liar.

Evidence handling. Evidence is logs, images, memory ideas, and disk ideas, plus the timeline you build. In a company, forensics may mean specialists and legal hold. In this course, you learn the habits: do not change what you still need to read; record the time you copied a log; store the copy where the suspect workload cannot edit it; write who handled it. Disk and memory images are specialist tools. Know that memory is volatile and that powering off can destroy it, and that a disk image is a copy you hash so you can show it did not change. This chapter does not give imaging commands for breaking into a machine. It tells you to call a prepared runbook and to practice copying logs in a lab you own.

Timeline. A timeline is a list, not a poster. Each line has a time, a source, and a fact. "10:14 PT, cloud audit, lab user attached a policy." "10:16 PT, runtime, shell in pod lab-pay." Sort them. Gaps are clues: a missing trail, a clock skew, a source you forgot to ingest. The timeline is the artifact Infosys-shaped clients and your own future self will believe. Memory will edit the story by morning.

Credential rotation at scale. One secret is easy. A platform has many. Keep an inventory: what it opens, who rotates it, how the app reloads it. During an incident, rotate by blast radius: the stolen key first, then the keys that sat beside it, then the sessions. A rotation that needs a full rebuild of a secret no one documented will stall the response. Practice one rotation in the lab until the app stays up, which Chapter 10 called rotation without downtime.

Isolation and quarantine. Isolation is network and identity. Quarantine the pod with a policy you tested in Chapter 21. Revoke the role. Disable the user. Leave the evidence process running if it is safe. Pulling the power is a choice when the spread is faster than the copy. Write that choice in the runbook so the midnight engineer is not inventing ethics and tactics together.

Rollback versus rebuild. Rollback returns a previous good version. Use it when the bad thing is a change you shipped and the platform underneath is still trustworthy. Rebuild when the host, the cluster, the runner, or the account may itself be modified. Rebuilding from the same compromised image is theatre. Immutable recovery means you deploy a digest you can justify, from a pipeline you have re-secured, onto infrastructure you replaced or re-imaged. Netflix-shaped speed still prefers rollback when it is honest. Honesty is the digest and the trail, not the urge to be green.

Clocks and handoffs. Write the next update time in the first note, and keep it even when the news is "we still do not know." Hand off the lead explicitly when the human must sleep. A tired lead who will not leave is how steps get skipped and evidence gets deleted. The runbook names the deputy. Swiggy-shaped peaks and Netflix-shaped nights do not pause because your best engineer is awake. The sequence pauses for no hero.

Communication. Say what you know, what you do not know, and when you will speak again. Do not name a cause you have not shown. Customers of a Paytm-shaped or BHIM UPI-shaped service need practical steps that do not ask them to hand a password to a stranger. Internal updates use the same facts. A second story in a side chat becomes the story the regulator hears.

Post-incident control. The review is short. What detected it, and how late. What contained it. What evidence was missing. Which control would have made this boring. Assign one control: a new detection, a narrower role, a joiner-mover-leaver fix, a retention change. Schedule it. A review that lists twenty ambitions and ships none is a speech.

Tabletops and runbooks. A tabletop is a conversation with a clock. You read a scenario, you do not attack a system. "A lab token appeared in a commit." Walk the checklist. Notice the missing phone tree, the missing log, the person who is on holiday. Fix the runbook. A runbook is one page per scenario: leaked secret, pipeline compromise, cloud key abuse, bad workload. Each page has the first five actions and the evidence to save. If the page needs a hero, rewrite the page. A drill that only talks, and never copies a log, has not been drilled. The copy is the point of the exercise, not the speech.

A one-page leaked-secret runbook. Lab wording.

1. Name the incident and the lead.
2. Revoke the secret. Record the time.
3. Rotate anything it could read.
4. Export audit for the secret's life. Do not delete the source yet.
5. Search the lab repo history and the log index.
6. Decide rollback or rebuild for anything that used the secret.
7. Write one new control before you close.

23.3 How.

1. Write "own lab only" and name an incident lead, a communicator, and a technical lead, even if all three are you.
2. Write three severity levels in five lines.
3. Create four one-page runbooks: leaked secret, bad pipeline, cloud key, bad pod.
4. Put the evidence export step above the delete step in each runbook.
5. In the lab, trigger a runtime alert on purpose.
6. Copy the alert, the digest, and the audit event to a folder the pod cannot write.
7. Quarantine the lab pod. Do not delete it until the copy exists.
8. Revoke the lab cloud session and record the time.
9. Start a timeline as a list of time, source, and fact.
10. Rotate one lab secret and reload the app without printing the new secret.
11. Search the lab log index for use of the old secret.
12. Freeze lab deploys for the drill. Then unfreeze only for a clean pipeline.
13. Rebuild the lab workload from a digest you recorded before the drill.
14. Confirm the healthy path returns and the bad process does not.
15. Decide in writing whether this drill was a rollback case or a rebuild case, and why.
16. Write a customer-style note for a fake lab user. Include what you know and what you do not.
17. Add one detection or one tighter role because of the drill.
18. Time the drill. Write the minutes next to the runbook.
19. Repeat the leaked-secret tabletop with a partner if you have one, or aloud if you do not.
20. Note one missing log. Add it to the Chapter 22 list.
21. Hash the exported evidence file and store the hash beside it.
22. Practice a trail-disabled alert and treat it as severity one in the lab.
23. Schedule the next tabletop. Put it on a calendar you will see.
24. File the lesson in the same repo as the runbook.
25. Stop. No ransom experiments, no wiping of systems you do not own, no "practice containment" on an employer or on the companies named in this course.

23.4 A real public incident.

Name. Cyber-attack on Norsk Hydro.

Date. Hydro said it was hit in the early hours of Tuesday 19 March 2019. Hydro published an update the same day describing isolation of plants and a switch toward manual operations, and further updates as recovery continued. Hydro later summarized the attack and the restoration on its own site.

Source. Hydro, "Update: Hydro subject to cyber attack," 19 March 2019, and Hydro's summary "Cyber-attack on Hydro," which states the date, the rebuild from backups, and an estimated cost of around 800 million Norwegian kroner. Public reporting the same week, including Reuters, said the Norwegian National Security Authority identified the malware as LockerGoga ransomware. This chapter does not describe how that malware spreads or how a ransom is paid.

What happened. Hydro, an aluminium producer, said an extensive cyber-attack hit operations across the company. Extruded Solutions took the hardest operational blow. Other business areas kept producing close to normal with manual workarounds. Hydro's public updates say it isolated plants and operations after detection, put safe operation first, and worked with internal and external experts. The later summary says PCs and servers were reviewed and cleaned, and encrypted machines were rebuilt from backups, under guidelines meant to keep the restoration safe. Hydro estimated the total cost at around 800 million Norwegian kroner and said it reorganized its security team to detect and respond better. Authorities, including Norway's National Security Authority, were part of the public record of the response. The lesson is sequence. Isolation and manual running kept a dangerous industrial business from improvising. Rebuild from backups beat a hope that encrypted machines could be negotiated back to life. Communication was public and repeated. Recovery was expensive and still orderly. A team that has never isolated anything will not invent this calm on the morning it needs it.

How to stop.

1. Name the lead. Isolate affected networks and workloads. Prefer tested quarantine over random power cuts, unless safety says otherwise.
2. Preserve logs and one copy of evidence before wide rebuilds erase the trail.
3. Keep people safe first in any system that moves the physical world, and keep a manual path if you have one.
4. Rotate credentials that may have been taken. Disable new privileged accounts you do not recognize.
5. Speak on a clock: what you know, what you do not, when you will update.

How to prevent it again.

1. Runbooks and tabletops for isolate, preserve, rebuild.
2. Backups that you have actually restored, stored where the ransomware-shaped blast cannot encrypt them too.
3. Immutable redeploy from known digests once the platform is trustworthy.
4. Detections and logging from Chapter 22 so identification is not a rumor.
5. A lessons review that changes one control, as Hydro's public account says the security team was reorganized to detect and respond.
6. Practice on a lab you own. Do not simulate ransomware on a company network.

This is for learning only, and only inside a system you are allowed to test.

23.5 A simple picture.

Prepare, identify, contain, eradicate, recover, and learn sit in a loop. Evidence is copied before the pod is replaced. The rebuild arrow leaves a known digest, not the drifted node.

23.6 A short animation.

A workload blinks as quarantined. The log copy stays bright. The replacement pod comes up only after the copy exists.

23.7 Conclusion.

Raju had a timeline of four lines and a lab app that came back from a digest he trusted. The shell pod was gone only after the copy. Guru-ji stacked the runbook on the log export. "You can survive a bad morning. Chapter 24 is how you prove the controls to someone who was not in the room, without burying the city in paper. Evidence from the pipeline and the cloud, mapped to a duty, on a clock."

• Response is a sequence. Evidence comes before deletion.
• Rollback when the platform is clean. Rebuild when it may not be.
• Secrets, pipelines, cloud accounts, and clusters each have a one-page runbook.
• A lesson that does not change a control is a story.
• Norsk Hydro, 19 March 2019, is the isolate, manual run, and rebuild-from-backup lesson.
• Own lab only.

END OF CHAPTER 23 MASTER CLASS