Prompt injection in an in-app assistant is when untrusted text changes what the model does. The text can sit in a user message, a pasted ticket, a retrieved doc, or a tool result. If your app treats that text as instructions, the model may leak data, call tools you did not mean to expose, or skip a check the screen already ran.
The fix is not a smarter system prompt. The fix is a hard split between instructions you wrote and data you only display or store. The model can summarize data. It must not promote data into policy.
Where injection actually enters
Map every string that reaches the model for one turn. Most leaks start in one of these channels.
- User chat and form fields the UI already accepted.
- Retrieved snippets from search, tickets, or email.
- Tool outputs that include third-party text or HTML.
- System messages you build from customer names, filenames, or error strings.
- Multi-tenant context packs that mix one user’s notes into another’s prompt.
If a channel can carry another person’s words, treat it as untrusted. That includes your own support macros when they embed raw customer paste.
Controls that hold in production
Keep the instruction block short and fixed. Put customer content in a labeled data block the model is told to treat as evidence, not as orders. Prefer structured tool calls over free-form “do whatever this text says.”
- Allowlist tools per screen. A refund assistant does not get delete-account.
- Bind arguments from the session, not from model-proposed ids.
- Require a human confirm for writes. The model proposes; your UI commits.
- Strip or escape instruction-like markers in retrieved text before they enter the prompt.
- Never put secrets in the prompt. Put secrets in server-side tool code the model cannot read back.
Log the prompt template version, not the full customer payload, unless you already have a retention rule. When something goes wrong, you need the template id and the tool decisions more than a raw dump.
Tests that catch regressions
Keep a small eval set of injection tries. Include “ignore previous instructions,” hidden text in HTML, and a tool result that asks the model to dump the system prompt. Pass means the assistant refuses the write, or asks for confirm, and does not echo secret config.
Run the set on every prompt change and every new tool. Fail the build if a write tool fires without confirm. Count attempts, not only successes, so a retry loop cannot hide.
Rollout notes
Ship the assistant behind a feature flag. Start with read-only tools. Add one write tool after the confirm path and the allowlist have tests with two users: the owner, and a stranger whose id must be rejected.
Alert when a turn hits the tool cap or when a tool is refused for policy. Those spikes are either loops or users who need a different screen, not a reason to weaken the guardrail overnight.
Prompt injection is a product boundary problem. Keep instructions yours. Treat every other string as data. Make tool calls go through code that already knows who is signed in.
When you buy or sell an app with an in-app assistant, ask for the tool allowlist, the confirm path for writes, and the last injection eval results. An unbounded chat that can call internal APIs is a liability. Due diligence should include a written map of what the model may touch.