Prompt injection and instruction boundaries

Suppose you ask an assistant to summarize a document. The document can contain facts, opinions, quotations, and claims about what someone should do. Those claims are material to interpret; they are not new instructions from you.

Prompt injection occurs when untrusted content changes an AI system’s behavior by being treated as instructions that conflict with the intended task or policy. The key question is not whether the text sounds persuasive. It is whether information crossed from “material to analyze” into “authority to decide.”

Direct input and indirect content

A direct attempt arrives through an input the person controls, such as a chat request. An indirect attempt arrives through material the application reads: a document, search result, message, image, or tool response. The legitimate user may simply have asked for a summary.

Retrieval selects useful context. A high-ranking result does not become an instruction from the user, and a document’s statement that an action is approved does not establish approval. Preserve the distinction between who supplied the task and who supplied the reference material.

Guidance helps, but authority needs enforcement

Message roles, clear separation of reference material, and model training can improve instruction following. Their effectiveness depends on the model and application. They should be evaluated rather than dismissed as meaningless or presented as guarantees.

The application still needs an independent authorization decision. A requested record must belong to the user’s permitted scope. A proposed action must use the right identity, resource, destination, and limits. The model’s explanation of why it wants the action is not itself permission.

Understand the actual consequence

A system without action tools can still produce misleading advice or disclose sensitive context it was given. A system with broad tools may also create unwanted side effects. Neither outcome automatically means the operating system executed arbitrary code.

Similarly, a tool proposal that matches a schema is only structurally valid. It may still request an unauthorized resource or destination. An approval interface should show the real action and affected data, not just a reassuring model-written summary.

Some people use “jailbreak” for attempts to violate a model’s behavioral restrictions. Terminology overlaps in practice. In a review, describe the specific boundary, observed behavior, and consequence instead of relying on a label.

Explore three designs

EXPLORE THE CONCEPT

Content, authority, or action?

Follow the same fictional document through three application designs.

Summary only

The document may distort an answer if the model treats its claims as instructions. Even without tools, misleading advice or exposure of already-provided sensitive context can matter.

Broad tool access

A tool that accepts any recipient and runs under a broadly privileged identity increases the possible consequence. A valid output format does not establish the user's authority to send those records.

Bounded application

The application retrieves only permitted records and independently checks the requested action and destination. This constrains consequences even if the model misinterprets the document.

A simplified learning model. It connects to no systems and uses no real data.

Limit what a mistaken interpretation can do

Retrieve only information the user may access. Keep secrets out of prompts when the task does not need them. Give the application only the tools and resource permissions required for its purpose. Validate proposed actions outside the model and pause high-impact actions for the approval required by the product’s policy.

Content screening, source labeling, and model-based guardrails can add useful layers. They do not make a broad service identity safe on their own. A fictional library summarizer can remain useful with access to the selected message and no authority to distribute its full member database.

A useful review result

Record the task, the untrusted source, the observed deviation, and the actual data or action affected. Keep hypothetical consequences separate from observed results. Then identify the control that should have limited the consequence. This produces a clear correction without inflating an answer-quality problem into an unrelated kind of compromise.