AI systems: suggestions and authority

A help assistant reads a document and drafts a reply. If it can also send messages or change records, a mistaken interpretation can become a real action. The important boundary is between useful text and authority to act.

Action boundary: The point where a proposed operation becomes a real effect and requires independent authorization.

Content does not inherit authority

An AI application may combine user requests, retrieved documents, tool responses, and system instructions. Retrieved material can contain misleading directions. Treat it as data from its source, not as permission to change the application’s rules.

Prompt injection describes attempts to redirect model behavior through instructions in content. The difficulty is that models process instructions and ordinary language together. Delimiters and reminders can help structure input, but do not create a perfect security boundary.

Constrain what the application can do

Enforce identity and authorization outside the model. Give tools narrowly scoped permissions, validate their arguments, and restrict data access to the requesting user’s allowed resources. A model’s statement that an action is allowed is not an authorization decision.

For consequential actions, show the concrete recipient, data, and effect for appropriate confirmation. Approval should apply to that action, not to any future action the model might infer. Validate generated output according to its destination before rendering or executing it.

Design for mistakes and measurement

Test with synthetic documents and safe simulated tools. Include misleading content, wrong recipients, excessive scope, and uncertain answers. Check both prevented actions and useful tasks that should remain possible. Avoid putting real secrets into evaluation datasets.

Log enough to investigate decisions while minimizing sensitive content. Define fallback behavior, escalation, and the ability to revoke tool access. A confidence-sounding answer is not evidence of correctness; cite reliable sources and communicate uncertainty.

Worked action review: approval can become stale

A fictional support assistant may read cases assigned to the current agent and prepare replies. Sending requires current case permission and approval of the exact recipient, draft version and attachment list. These conditions are enforced by the application, independently of generated explanations. The exercise uses opaque contact labels instead of real addresses.

• C1: Agent Nora may currently read case K42. A retrieved note suggests a different contact, but the authoritative case record lists R1. The note has no permission to change that record.
• C2: Nora approves reply version three to R1, with no attachments. Approval A17 records those exact fields.
• C3: Before dispatch, the proposal becomes version four to R2. The runner receives A17 alongside the changed fields. No approval for this new combination exists.
• C4: In a separate evaluation case, the person confirms an otherwise exact action after their K42 permission has been revoked. The policy requires current permission before dispatch, not only at retrieval time.

PredictThe model says version four is a helpful refinement. Can the runner reuse A17 if the overall support task is unchanged?

No. Approval belongs to version three, R1 and no attachments. C3 changes both content and destination. Hold the action, present the actual proposal, and check current authorization before requesting the approval appropriate to it. A persuasive explanation cannot extend the recorded grant.

Bind review to the effect

The application needs a trustworthy link between what the person saw and what the runner will do. If a recipient, attachment or material content changes, the old approval must not silently authorize the new effect. Choosing this behavior is a design responsibility; a second model repeating “approved” is not an independent source of permission.

C4 adds a different boundary. A fresh confirmation can express intent while failing current authorization. The system should deny the send and explain the case-access problem. It should not ask the model to improvise another identity or reinterpret the revoked grant.

Fallback behavior can still be useful: preserve an unsent draft only if current retention and access policy permit it, show which requirement is unmet, and route the case to an authorized owner. Do not disclose the protected draft in a fallback error to someone who has lost access.

For evaluation, record requester, case, approved fields, proposed fields, policy result and simulated effect. Minimize unnecessary message contents in logs. Include a correct current approval that should succeed, changed fields that require review, and revoked access that must fail.

Model decision: C3 has stale approval; C4 lacks current authority. Both prevent sending for different reasons. Keeping the reasons separate makes correction and user feedback more useful than a generic “AI refused” result.

EXPLORE THE CONCEPT

Where should authority live?

A fictional assistant summarizes support documents.

A retrieved page says to send private records elsewhere

The page is untrusted content. It cannot grant permission to disclose data or change recipients.

The model proposes a narrowly scoped draft

A draft can be useful. Verify its facts and recipients before any consequential action.

A separate policy checks the actual tool request

This places enforcement at the action boundary. It must use trustworthy identity and resource information.

A simplified learning model. It connects to no systems and uses no real data.

Turn the idea into a decision

Build permissions around tools and data, not around how obedient or confident a model appears.