Chapter 31: AI, LLM and MLOps Security

31.1 Why.

Welcome to Chapter 31.

I am Ravindra Bagale.

Raju had no engineering degree. He had a chat box wired to a model, a folder of documents, and a tool that could query the lab database because that felt helpful. He pasted a customer complaint into the prompt, then pasted a system design, then asked the model to "use whatever tools you need." Guru-ji unplugged nothing. He took the tool list away.

"The model is a brilliant, untrusted parser," Guru-ji said. "It will follow the loudest instruction it can see, including instructions hidden in a document you retrieved. It is not an employee. It does not get a production role. It does not get to decide that a secret is fine to print."

Raju said the vendor promised safety. Guru-ji said vendors are part of the system, not the whole system. "Suppose we are Netflix and a support assistant can see one viewer's account, and a retrieved article tells it to summarize every account. Suppose we are Swiggy or Zomato and an order note is untrusted text that tries to change a refund tool. Suppose we are Paytm or PayFast and a model can call a payment API with arguments it invented. Suppose we are Amazon and a training bucket is world-readable because the experiment was temporary. Suppose we are WhatsApp-shaped or BHIM UPI-shaped products and prompts contain personal data that a log retains forever. Suppose we are Infosys and the client asks where the model runs, what data it was allowed to see, and how you test that a document cannot widen that allowance. Fluency is not authorization."

31.2 What.

A threat model for learning systems. Draw four assets: the data, the model artifact, the pipeline that trains or evaluates, and the application that calls the model. Threats differ. Data can be poisoned, stolen, or retained too long. Artifacts can be swapped. Pipelines can leak secrets the way any pipeline can. Applications can be talked into misusing tools. Write the abuse you will not allow: moving money, deleting accounts, revealing other tenants' data, or executing code the model wrote. If a tool can do that, a prompt must not be sufficient to call it.

Supply chain of models and data. A model file is an artifact. Pin it by digest. Store it in a registry with access control and a promotion rule: dev, eval, prod. A notebook download from a random forum is not a promotion. Training data needs provenance. Who collected it, under what right, and what was excluded. A dataset that accidentally includes secrets or real customer dumps is a breach you trained into the weights. Synthetic lab data is the default here. Track a model SBOM in spirit: base model, fine-tune data version, code version, and evaluation result. You cannot patch what you cannot name.

Registries and MLOps pipelines. The pipeline that trains, evaluates, and promotes is a production pipeline. Protected branch. Isolated runners. No long-lived cloud key in the notebook. The training role can read the training bucket and write the model registry. It cannot read the production customer store "to make the demo better." Promotion to production is a signed, reviewed step. Rollback is a previous digest, practiced. Chapter 26's golden path should include an ML path so teams do not invent a shadow GPU account.

Prompt injection, named not demonstrated. Untrusted text can contain instructions. That text might be a user message or a document your retrieval layer fetched. The failure mode is that the model treats the text as orders and tries to ignore the application's rules, reveal a hidden prompt, or call a tool. You need to know the name and the impact. You do not need a catalog of bypass strings, and this chapter will not give you one. Defenses are architectural. Separate instructions from data in your application design. Retrieve only documents the user may already see. Do not put secrets in the prompt. Treat model output as untrusted data: encode it before it reaches a page, and never pass it to a shell or a payment call without a deterministic check your code performs. Test with a harmless canary you invented, such as a fake word that must not appear, and assert the app refuses or strips the attempt. Do not test against a vendor you do not own. Do not try to jailbreak production systems.

Leakage and privacy. Prompts and retrieved text are data. Classify them. Do not send payment data, message contents, or customer secrets to a model endpoint that your policy forbids. Vendor processing is a supplier question from Chapter 24: where is it stored, for how long, is it used to train, who can see it. Log the fact of a call, the user, the tool names, and a hash or id of the prompt if you must investigate. Do not log the raw prompt when it holds personal data. Retention on AI logs is a product decision, not an accident of disk.

Tools, plugins, and agents. An agent is a loop that can call tools. Each tool is a permission. List tools explicitly. A refund tool in a lab should move a synthetic balance only, with a maximum amount enforced in code, not in the prompt. The model may request a call. Your policy code decides. Human confirmation belongs on actions that are hard to undo. Least privilege from Chapter 18 applies to the identity the tool uses. A tool role that can read all orders will be asked, by some string somewhere, to read all orders. Do not rely on politeness.

Retrieval and access control. Retrieval-augmented designs pull text into the prompt. The pull must apply the same authorization as the rest of the product. If the user cannot open the document in the app, the retriever must not fetch it. Poisoning means someone placed misleading text where you will later trust it: a public page, a ticket, a knowledge article. Prefer sources you govern. Separate "quoted data" from "orders to the app" in the way you present context. Re-index when you delete a user's data. A vector store is a data store from Chapter 28. Encrypt it, restrict it, and do not leave embeddings of production text in a world-readable bucket.

Output handling. The model may return HTML, SQL-looking text, or a confident wrong total. Render it as text. Parameterize any query yourself. Recalculate money in code. Show citations to documents the user may see, not to hidden system text. A beautiful sentence can still be the wrong control.

Evaluation and safe testing. Evaluate quality and evaluate abuse resistance as separate suites. The abuse suite uses your own fixtures: a canary secret, a document the test user must not retrieve, a tool the test must not successfully call. Run it in CI against your lab app. Record failures as tickets. This is not a red-team procedure against a public chatbot. It is a regression test. NIST's AI risk framework, as awareness, asks you to govern, map, measure, and manage. You can practice that loop on one lab assistant without claiming a certification.

Identities and guardrails. The workload that calls the model uses a workload identity. Rate-limit the calls so a loop cannot become a bill and a leak. Filter obvious secret patterns outbound if your policy requires it, and still do not put secrets in the prompt. Guardrails are checks around the model. They are not the authorization system. If the guardrail fails open, the tool must still be impossible to invoke without policy code.

A teaching sample: a tool gate. Lab only.

tool: lab_refund
max: 100 units of synthetic credit
allow: the signed-in lab user only
enforce: application code checks user, amount, and idempotency key
model: may request, may not approve
log: user id, tool name, allow or deny; no raw complaint text

Seminar notes. Raju wanted a longer system prompt and called it a control. Guru-ji called it a wish. Instructions inside the model are not a boundary. Boundaries are code: which documents can be retrieved, which tool can be called, which amount can move, which log line can be stored. A longer prompt can still be contradicted by text you retrieved. Treat that as a known limitation. Do not collect bypass examples to prove you are clever. Prove the boundary with fixtures. A canary must not appear. A forbidden document must not be fetched. A tool must not act for a user id the model supplied. Those three tests belong in CI.

Model theft and extraction, as awareness, means someone may try to copy a model you host by querying it heavily, or may simply download an artifact you left in a public bucket. You do not practice extraction against a vendor. You lock the registry, you rate-limit the endpoint, and you keep the production digest private. Watermarks and legal terms are outside this lab. Access control is inside it. The same bucket policy as Chapter 28 applies to training sets and to weights. A world-readable experiment bucket is a publication.

Training-data integrity is provenance plus access control. Record the dataset version next to the model digest. If you cannot rebuild the evaluation, you cannot explain a regression. Poisoning, in practice, is untrusted text entering a place you later trust: a ticket corpus, a public page, a shared drive. Prefer governed sources for anything the assistant treats as policy. Customer complaints are data, not policy. Retrieve them as quoted text for the signed-in user only. Re-index when you delete a user so embeddings do not become a shadow copy that ignores retention. A vector store is a database. Give it a small role.

Insecure output is a rendering problem and a business-logic problem. The model may suggest an HTML fragment or a confident total. Your page treats the fragment as text. Your ledger recomputes the total. Never pipe model text into a shell, a template with code execution, or a database statement built by concatenation. That rule is the injection defense from earlier chapters, applied to a fluent source. Logging and privacy sit beside it. Store tool decisions and correlation ids. Do not store raw prompts that contain personal data unless a written purpose and a short retention say so. Vendor retention is a supplier question. If the lab policy says data must not leave the machine, use a local fixture model or do not call out. Infosys-shaped client data does not go into a personal chatbot account because the chatbot was convenient. That convenience is an untracked processor.

NIST's AI risk work, as awareness, is a loop: govern the use, map the harms, measure with tests, manage with kill switches and tickets. You can run that loop on one lab assistant in an afternoon without claiming a certification. Evaluation for abuse resistance is the fixture suite, repeated when the prompt, the tools, or the model digest change. A red-team engagement against your own lab app, if you ever run one, uses the same charter as Chapter 34: scope, synthetic data, no recipes in the notebook, output is a regression test. It is not a contest to defeat a public service. Paytm-shaped or PayFast-shaped money movement stays behind code, with a human on irreversible steps. The model may draft a sentence. It may not approve the movement.

One more pass. Guardrails that fail open are monitoring, not control. If the filter service is down, the tool call should stop, not proceed. Record that choice on the objective card from Chapter 27, because a fail-closed assistant may burn error budget and a fail-open assistant may burn trust. Prefer failing closed on tools and failing soft on prose, with a clear message. Secrets and cloud identities for the training job stay on the training bucket. The job that serves user traffic does not need the training role. Two roles, two digests, one registry with promotion. A model card in the lab can be ten lines: data class, digest, evaluation date, known limits, owner. That card is architecture, small enough to keep true.

31.3 How.

1. Draw data, model, pipeline, and app as four boxes for a lab assistant.
2. Use only synthetic documents and synthetic users.
3. Put the model artifact in a lab registry and record its digest.
4. Give the training or calling role access to one bucket, not the account.
5. Remove any secret from the prompt template and from the notebook.
6. List tools. Delete every tool that is not required for the lab.
7. Implement the refund-style tool so code enforces the user and the maximum.
8. Prove with a unit test that a model-supplied user id cannot override the signed-in user.
9. Add retrieval that filters by the caller's allowed documents.
10. Add a fixture document the caller must not see, and assert it is not retrieved.
11. Add a canary string in a system-side fixture and assert it never appears in the answer or the log.
12. Render model output as text in the lab UI, not as raw markup.
13. Set retention on lab prompts to a short window, or do not store them.
14. Log tool decisions without raw personal text.
15. Rate-limit the lab endpoint.
16. Promote a model digest only from the pipeline, with a note of the evaluation result.
17. Roll back to the previous digest once, as a drill.
18. Map one control to a govern-or-measure style objective in a single line.
19. Put the assistant on the service catalog with owner and data class.
20. Review the vendor or local runtime: what is retained, and whether your lab data is allowed there.
21. Stop. Do not jailbreak, scrape, or test systems at Swiggy, Paytm, Amazon, BHIM UPI, WhatsApp, Zomato, Netflix, Infosys, PayFast, or any model endpoint you do not own.

31.4 A real public incident.

Name. ChatGPT outage and cross-user data exposure on 20 March 2023.

Date. The incident occurred on 20 March 2023. OpenAI published "March 20 ChatGPT outage: Here's what happened" on 24 March 2023.

Source. OpenAI's blog post of 24 March 2023. OpenAI described a bug in an open-source library around Redis that could, during a window, show some users other users' chat history titles and, for a smaller set, payment-related information of subscribers. ChatGPT was taken offline while the problem was addressed. This chapter does not describe how to reproduce the bug.

What happened. A conversational product feels like a private room. Isolation between users is the room's wall. A defect in a shared cache layer made some of another person's history visible. The impact was personal data and, in a limited slice, billing data. The operational response was to take the service down rather than leave the wall missing, then fix and explain. For builders, the lesson is that an LLM feature is still a multi-tenant application. Caches, logs, and prompts are data stores. A library bug in the session layer can breach tenants without anyone "tricking" the model. Threat models that only talk about clever prompts miss the ordinary failure. Test tenant isolation the way you test payments. Prefer an outage to a leak when you discover the wall is gone. Do not log one tenant's prompt into another tenant's trace.

How to stop.

1. Disable the feature or the cache path that is mixing tenants.
2. Preserve evidence for the investigation without copying personal prompts into a new leak.
3. Identify who was affected and what classes of data crossed the wall.
4. Patch the component, verify isolation with fixtures, then restore service.
5. Notify people as the duty requires. Rotate or invalidate exposed billing artifacts if your case includes them.

How to prevent it again.

1. Test that user A cannot read user B's history, including titles, embeddings, and logs.
2. Treat the session store as production data with least privilege and encryption.
3. Keep AI logs tenant-scoped and short-lived.
4. Pin and review libraries that sit under the session layer.
5. Practice a kill switch that takes only the assistant down.
6. Do this in a lab you own. Do not probe a public model for other people's chats.

This is for learning only, and only inside a system you are allowed to test.

31.5 A simple picture.

The user speaks. Retrieval is filtered. The model may ask for a tool. Policy code allows or denies. Output is untrusted text.

31.6 A short animation.

A tool request appears. The policy stamp says deny. The synthetic balance does not move.

31.7 Conclusion.

Raju's lab refund could not exceed the maximum, and the hidden fixture never entered the prompt. Guru-ji let the tool list stay short. "In Chapter 32 we return to delivery. Git becomes the desired state. A controller applies it. Progressive delivery lets a bad digest die before the whole city takes it."

• Models and datasets are artifacts with owners, digests, and promotion rules.
• Untrusted text can try to redirect the app. Do not demonstrate bypasses. Do enforce tools in code.
• Tenant isolation includes caches and titles, not only clever prompts.
• OpenAI's 24 March 2023 account of the 20 March outage is the shared-cache lesson.
• Own lab only.

END OF CHAPTER 31 MASTER CLASS