Chapter 22: Observability, Logging, SIEM, Detection Engineering

22.1 Why.

Welcome to Chapter 22.

I am Ravindra Bagale.

Raju had no engineering degree. He had a disk that filled with logs and a hope that fullness was safety. Guru-ji asked a harder question. "If a payment failed at 10:14, can you show the request, the pod, the cloud role, and the deploy that touched that route, on one page, without guessing?" Raju could show four tools and no page.

"Observability is how you see the service," Guru-ji said. "Logging for security is how you see the abuse. A SIEM is where those lines meet rules. Detection engineering is the job of writing rules that a human can act on, then tuning them so the human does not quit. Suppose we are Paytm and a balance read has no correlation id, so the runtime alert from Chapter 21 cannot find the customer path. Suppose we are Swiggy and the gateway log lives seven days while the investigation needs seven months. Suppose we are Amazon and a trail that records admin calls is turned off and nobody alarms. Suppose we are Netflix and a dashboard shows errors but not who assumed a role. Suppose we are Infosys and the client audit asks for evidence that privileged changes were detected, and the answer is a screenshot of a green cluster. Suppose we are BHIM UPI, WhatsApp, Zomato, or PayFast in the sense of a busy system whose logs are unstructured paragraphs. A paragraph does not join. A field joins."

An alert that arrives four years after the first quiet access is not a tool. It is a late letter. This chapter is how you refuse to write that letter.

22.2 What.

Logs, metrics, traces, and events. Logs are records of things that happened. Metrics are numbers over time: latency, error rate, CPU. Traces follow one request across services. Security events are logs you are willing to alert on: a login, a deny, a role change, a shell in a pod. You need all four. Metrics tell you the bill and the pain. Traces tell you the path. Logs tell you who. Events tell you to stand up. A platform that has pretty latency charts and no admin audit is a cockpit without a door sensor.

OpenTelemetry. OpenTelemetry is a common way to emit traces, metrics, and logs with shared context. Use it so a request id born at the gateway arrives in the service and in the runtime alert. You do not need every feature. You need one id that survives the hop. Vendor lock on a private format is how you fail the export the day you change tools. Keep the semantic habit even in a small lab: trace_id, subject, action, result.

Central logging. Applications must not be the only shelf. Nodes die. Attackers delete local files. Ship logs off the box to a place the workload cannot erase. Separate the right to write telemetry from the right to delete it. A Paytm-shaped payment log and a Swiggy-shaped order log can share a platform and must not share a bucket policy that lets every developer empty the archive. Index what you search. Archive what you must keep cheaply. Encryption in transit and at rest is the default, with keys from Chapter 10's discipline, not a new invention.

Structured logs and correlation ids. One JSON object per event. Time in one zone, stored with an offset, displayed to humans in Asia/Kolkata when the team lives there. Stable field names. No secrets, no full tokens, no full card numbers. A correlation id on the request. A user or workload subject when you have one. A result. If you log a free-text line, you will parse it with regret. Structured now is a gift to the SIEM later.

{"time":"2026-10-04T10:14:00+05:30","trace_id":"lab-1","subject":"lab-user","action":"order.read","object":"order-9","result":"deny","cluster":"lab"}

That line can join to a cloud deny and to a runtime alert. A sentence in a file called app.log cannot.

Security logging requirements. Decide, in writing, the events you always keep: authentication successes and failures, privilege grants, policy changes, secret reads if the platform can log them without logging the secret, admin API calls, pipeline approvals, Kubernetes audit for exec and role binding, and the runtime alerts from Chapter 21. "Log everything" is how disks fill and how secrets leak into the log. "Log nothing sensitive" is how you cannot investigate. Name the events. Redact the values that would turn the log into a second copy of the vault.

SIEM concepts. A security information and event management system collects, stores, and queries those events, and runs detections. Elastic, Splunk, and the cloud-native security lakes are implementations. The concept is the same. Ingest. Normalize field names so subject means subject in every source. Retain. Search. Alert. Case. If every team keeps a private grep folder, you do not have a SIEM. You have souvenirs. Start in the lab with one index and three sources: app denies, cloud audit, and runtime alerts. Resist a tool tour that ingests nothing you can explain.

Detection engineering lifecycle. A detection is a small product. Choose a behavior you care about, such as a human creating a new admin or a pipeline skipping a gate. Write the rule against fields you really ingest. Test it with a lab action you perform yourself. Measure false positives for a week. Tune. Ship behind review. Attach an owner and a runbook link. Retire the rule when the behavior cannot happen anymore. A rule nobody owns will page forever or never. Both are failures. Version rules in Git next to the code.

Sigma, as a concept. Sigma is a generic signature format for logs, so an idea can be shared and then compiled to a specific SIEM. You can sketch the idea even if you never install the compiler. The value is the fields and the false-positive note, not the brand.

# lab sketch — a privileged cloud change you caused yourself
title: Lab admin policy changed
logsource: {product: cloudtrail, service: cloudtrail}
detection:
  selection:
    eventName:
      - PutUserPolicy
      - AttachUserPolicy
  condition: selection
level: high
falsepositives:
  - the lab deploy role, which you exclude by a field, not by muting the rule

Translate the same idea to whatever your lab account actually emits. Do not paste a hundred community rules into production on day one. Each one is a promise to investigate.

Tuning and false positives. A false positive is an alert that is not the bad thing. A false negative is the bad thing with no alert. Tuning is how you trade them honestly. Exclude a known deploy role by an identity field. Do not exclude a whole account. Do not raise the threshold until the page stops if that also hides the theft. Write the exception with an expiry. Review it. Netflix-shaped noise at peak and Swiggy-shaped noise at meal time are predictable. Your rule should know the calendar without ignoring the outlier principal.

Cloud, Kubernetes, and CI audit. Cloud audit logs say who called which API. Turn them on in every account, including the lab, and alarm if they stop. Kubernetes audit says who exec'd into a pod, who bound a cluster role, who created a secret. CI audit says who changed a protected branch, who approved a deploy, who disabled a test. These three plus the app log are the minimum join for a DevSecOps incident. Chapter 11's pipeline and Chapter 18's assume-role should be visible here. If the deploy is invisible, the detection program is a story.

Alert routing and on-call. An alert names a severity, a service, and a first action. Page a human for the few that need minutes. Ticket the rest. Do not page the whole company for a lab info event. Do not send high severity only to a mailbox nobody reads. Test the route with a drill. Include the correlation id in the page so the human does not start from zero. Quiet hours do not apply to "trail disabled" or "break-glass used."

Threat hunting. Hunting is a search without a page, led by a question. "Did any workload read the secret store outside a deploy window?" "Did any token audience get rejected a thousand times?" Keep hunts short, write down what you looked for, and turn a repeated hunt into a detection. Hunting is not permission to wander through personal data for curiosity. Scope the query. Record it. Infosys-shaped evidence includes the hunt log, not only the incident log.

MITRE ATT&CK mapping. ATT&CK is a public catalog of tactic and technique names: initial access, persistence, credential access, discovery, lateral movement, collection, exfiltration, impact, and many techniques under them. Map your detections to tactic names so you can see a gap. If you have ten rules for miners and none for new cloud users, the map shows off. Mapping is a coverage tool. It is not a claim you are safe, and this course will not walk technique procedures. Name the column you cover. Write a rule for the gap. Do not collect technique ids the way tourists collect magnets.

Dashboards and KPIs. A security dashboard for this course is small. Time to detect a lab admin change. Percent of accounts with trails on. Number of unmuted high rules. Median time to close a true lab alert. Pipeline gate failures. Patch age belongs to a later conversation. Do not decorate a wall with counts of low findings. Leadership will optimize the count. Show time and coverage.

Integrity and retention. Retention follows the event. Authentication and admin events live longer than debug lines. Write the number in the standard: for example, admin audit for a year in the lab policy you choose, and say why. Protect the archive from the people who are most interesting to investigate. Hash or sign exports if you must show them later. Clock sync matters. A timeline with drifting clocks is fiction. Document the zone. When you show a timestamp to a teammate in Asia/Kolkata, convert it and label it.

Cases. An alert that repeats should become a case with a status: new, confirmed, false positive, or fixed. The case stores the query you ran and the control you changed. Without that, Chapter 23's timeline starts from chat fragments. A lab case can be a markdown file next to the rule. The habit is the point. The product brand is not.

A join you can demo. Perform one lab action: attach a policy using a lab admin. You should see the cloud audit event, the CI job if a pipeline did it, and no application panic. The detection fires once. The page contains the principal and the event name. That demo is worth more than a tool comparison.

22.3 How.

1. Write "own lab only" and list three sources you will ingest: app, cloud audit, runtime.
2. Turn on cloud audit logs in the lab account and confirm a new event arrives when you make a call.
3. Turn on Kubernetes audit for exec, role binding, and secret access if you run a lab cluster.
4. Turn on pipeline audit for approvals and for skipped jobs.
5. Change one app log from a sentence to a JSON object with time, subject, action, object, and result.
6. Add a correlation id at the gateway and log it in the service.
7. Redact tokens and secrets. Search the lab log for a sample password and prove it is absent.
8. Ship the three sources to one lab index the workload cannot delete.
9. Normalize one field name so the principal is called subject in all three.
10. Set a retention number for admin events and a shorter number for debug.
11. Alarm if the audit trail stops or the shipper falls behind.
12. Write one detection for a privileged cloud change you can trigger safely.
13. Trigger it. Confirm one alert, not fifty.
14. Add an exclusion for the known lab deploy role, with an expiry note.
15. Sketch the same detection as a Sigma-style rule in Git.
16. Map that rule to one ATT&CK tactic name, such as privilege escalation or persistence.
17. Write the gap beside it: a tactic you do not cover yet.
18. Route the high alert to a channel you actually watch. Route low alerts to a ticket list.
19. Run one hunt: principals who failed authorization more than a threshold you choose.
20. Turn that hunt into a ticket or a rule if it is worth repeating.
21. Build a tiny dashboard: alerts today, trail health, median time to see the lab event.
22. Sync clocks. Label times you show to people in Asia/Kolkata.
23. Export one day of admin logs to a second bucket and restrict delete on that bucket.
24. Review a week of false positives and tune one rule without muting the account.
25. Stop. No mass collection of other people's logs, no rules aimed at employers or at the companies in this course unless you are their employee following their law. The lab is yours.

22.4 A real public incident.

Name. Marriott Starwood guest reservation database security incident.

Date. Marriott said that on 8 September 2018 it received an alert from an internal security tool about an attempt to access the Starwood guest reservation database. Marriott said it learned during the investigation that there had been unauthorized access to the Starwood network since 2014. On 19 November 2018 the investigation determined the database was involved. Marriott announced the incident on 30 November 2018.

Source. Marriott International news release, "Marriott Announces Starwood Guest Reservation Database Security Incident," 30 November 2018. This chapter uses that release. It does not describe intrusion methods.

What happened. Marriott said the database held guest information for reservations at Starwood properties on or before 10 September 2018. Marriott believed the database contained information on up to approximately 500 million guests. For approximately 327 million of them, the information could include some combination of name, mailing address, phone number, email, passport number, account information, date of birth, gender, arrival and departure information, reservation date, and communication preferences, and for some, payment card numbers and expiration dates. Marriott said an unauthorized party had copied and encrypted information and taken steps toward removing it, and that on 19 November 2018 Marriott decrypted that set and saw that it came from the reservation database. Marriott reported the incident to law enforcement and began notifying regulators and guests. The detection lesson is the calendar. An alert fired on 8 September 2018. The access, Marriott said, reached back to 2014. A tool existed. A timeline of four years means logs, ownership, and investigation did not meet. Encryption statements were revised in later updates, which is a reminder to cite the primary release and to expect investigations to correct details. The control failure that matters here is not a missing chart of latency. It is security telemetry that did not become a decision while the access was still young.

How to stop.

1. Isolate the database and the paths into it. Preserve the logs you still have, including the alert that finally spoke.
2. Rotate credentials and keys that could read or decrypt the data.
3. Build a timeline from retained audit, not from memory.
4. Notify guests and regulators with facts you can stand behind, and correct the record if later forensics change the facts.
5. Keep the archive immutable while the investigation runs.

How to prevent it again.

1. Security events defined in advance, shipped off the system, retained long enough to see a slow intrusion.
2. Detections on bulk reads and on unusual access to reservation-shaped data, owned by a person who must answer.
3. Correlation ids and structured fields so an alert joins to a principal.
4. Alarms when logging stops. A dark window is itself an incident.
5. Hunts that ask "what did this identity touch in 90 days?" while the logs still exist.
6. Practice the join in a lab you own. Do not go looking through hotel or payment systems that are not yours.

This is for learning only, and only inside a system you are allowed to test.

22.5 A simple picture.

App logs, cloud audit, Kubernetes audit, pipeline audit, and runtime alerts land in one store the workload cannot delete. A detection reads fields, not paragraphs. A page goes to a human with a runbook. A hunt writes the next rule.

22.6 A short animation.

A privileged change happens. The rule pulses once. A muted rule does not pulse. Own the rule that still moves.

22.7 Conclusion.

Raju could point at one JSON line and one cloud event and say they were the same minute. Guru-ji nodded. "Seeing is not finishing. Chapter 23 is what you do when the detection is true: contain, keep the evidence, rebuild from what you trust, and write down what the control failed to do."

• Logs, metrics, traces, and security events answer different questions. You need the events.
• Structure and a correlation id are how joins happen.
• A detection is a product with an owner, a test, and a false-positive note.
• Retention and an alarm on a silent trail are part of the control.
• Marriott's 30 November 2018 announcement is the long-dwell lesson: an alert in 2018, access since 2014.
• Own lab only.

END OF CHAPTER 22 MASTER CLASS