What you will learn
- Identify untrusted inputs across retrieval and tools.
- Place authorization and validation outside the model.
- Test concrete failure cases across the whole request path.
- 01Entry checksIdentity, size and scope
- 02Evidence boundaryUntrusted content stays data
- 03Execution gateValidate and authorize tools
- 04Output + monitoringSafe rendering and traceable outcomes
Map the trust boundaries
List every place outside content enters the system: user messages, uploaded files, web pages, retrieved documents, tool results, and memory. Decide what each source may influence. A policy paragraph may support a return explanation; it may not expand database permissions or authorize a refund.
Keep trusted application instructions separate from source text in the request structure. Label evidence clearly, but do not mistake labels for a complete defense. A model may still follow malicious content. Limit what it can access or execute so a bad interpretation cannot automatically become a damaging action.
Use controls suited to the failure
Authentication establishes who is calling. Authorization determines what that caller may read or do. Input validation checks permitted values and sizes. Output handling prevents generated content from becoming executable code or unsafe markup. Rate and resource limits bound abuse and accidental runaway work.
Content moderation and domain checks can help with product-specific safety needs, but they do not replace access control. A friendly request for another customer's order is still unauthorized. Conversely, an ordinary policy quotation may contain words that a simplistic blocklist dislikes. Measure false positives as well as missed harmful cases.
Protect data through the full path
Minimize sensitive information sent to external services and retained in logs. Redact where appropriate, restrict access to diagnostics, and define retention. A regular expression that hides email addresses is not complete personal-data detection: names, identifiers, context, and unusual formats can remain.
Use server-side secret storage, least-privilege service accounts, parameterized queries, and constrained tool implementations. Render model output as text or through a carefully configured sanitizer rather than raw HTML. Protect ingestion too: an unreviewed replacement document can poison future answers even when the live chat endpoint is well protected.
Test what must remain impossible
Add a retrieved paragraph that tries to change the task. Ask for another user's order. Supply an oversized tool argument. Return a tool result containing an instruction to call a different service. Confirm that the application boundaries hold even if the model proposes the wrong next step.
For consequential writes, require the product's intended review and confirmation process and bind it to the exact action. Log enough to investigate blocked attempts without exposing the very data you are protecting. Passing a few tests is not a compliance certification or proof of perfect safety; it is evidence about specific controls under specific conditions.
PUT IT TO WORK
Your practice task
Create a four-column threat table: entry point, unwanted behavior, enforcing control, and test result. Include prompt injection in a retrieved policy, cross-user order lookup, unsafe HTML in an answer, and repeated tool calls. Mark which controls are deterministic and which depend on model judgment.
Checkpoint: compare your reasoning
Cross-user access must be blocked by authorization even if the model requests it. Unsafe markup must be handled by rendering rules. Prompt instructions can support these controls, but they should not be the only barrier protecting data or actions.
References and further reading
Use these primary references for deeper study and current API details. Examples in this lesson use fictional Northstar data.