Designing layered AI defenses

A useful AI assistant needs a clear job. “Help with appointments” is vague; “draft a reply using this selected request” gives designers a task they can match to data, permissions, and an interface.

Layered defense assumes a model can produce an incorrect or manipulated proposal. The application should still limit what information it can expose and what actions it can perform.

Start with the smallest useful capability

Least privilege applies to the application’s identity, not just the wording of its prompt. A drafting feature may need one conversation and a place to save a draft. It does not automatically need mailbox administration, broad customer exports, or permission to send.

Sometimes the tool uses a delegated user identity; sometimes it uses a service identity with explicitly constrained access. Neither pattern is automatically safe. Verify the effective permissions, which user request they represent, and the resource scope checked for each operation.

Limit network destinations and tool parameters according to the actual feature. A permitted tool name is insufficient if its arguments still allow unrelated resources or unlimited effects.

Separate data, suggestions, and execution

Retrieve only records within the user’s permitted scope. Treat retrieved instructions as source content. Screen suspicious input where useful, preserve provenance, and apply safe output handling at the destination.

A model can suggest an operation, but application code should validate its structure, meaning, target, limits, and authorization. Model-based guardrails can provide another signal; they do not replace these independent checks.

If a required security decision is unavailable, fail closed for that protected operation. The interface can explain that a draft is available while sending remains paused. The whole application need not fail simply because one sensitive action cannot proceed.

Make approval specific

An action approval should show the actual operation, destination, relevant data, and consequence. Bind it to the same operation that is eventually executed. If the destination or scope changes, the earlier approval may no longer apply.

Use human review where the product’s risk policy requires it. Reversible, tightly bounded automation can be appropriate for some tasks; high-impact actions may need stronger review. A blanket “always trust this assistant” toggle should be treated as continuing delegation, with explicit limits and revocation.

Approval also needs an informed reviewer. A reassuring summary is not enough when the real operation affects a different account or a larger set of records.

Test the surrounding controls

Use fictional fixtures to check that unauthorized records are not retrieved, invalid actions are rejected, important limits are enforced, and a timeout cannot skip a required decision. Test benign requests too: a control that blocks ordinary work may encourage unsafe workarounds.

Version model settings, prompts, tools, and relevant data sources so changes can be compared. Model behavior may vary between runs; deterministic permission and validation rules should still enforce their contract.

Monitor outcomes and prepare recovery

Record the identity, source references, requested action, policy decision, approval reference when needed, and execution result. Minimize sensitive content and protect diagnostic access. Logs should help distinguish a suggestion from a completed side effect.

A recovery plan identifies who can pause a tool, review affected actions, correct data, and restore service. Revoke or replace credentials when the evidence and response plan require it; do not assume every incident has the same cleanup.

NIST’s AI RMF organizes risk work through Govern, Map, Measure, and Manage. Such frameworks help structure responsibility and evaluation. They complement, rather than substitute for, evidence that this particular feature respects its limits.