AI on offense

AI can help people summarize notes, classify events, search documentation, or draft explanations. It can also be part of the system being protected. Both uses benefit from clear evidence, bounded access, and a way to correct mistakes.

This module considers the full application: the model, data sources, identity, user interface, optional tools, and operational processes. Some risks resemble familiar application weaknesses; others concern learned behavior, training data, or model evaluation. Understanding one category does not remove the others.

Use an assistant with evidence

A Language model can generate useful explanations without guaranteeing accuracy. Different model families use different training objectives; a conversational generator is not the definition of all AI.

For study, compare an explanation with the actual reference, ask for uncertainties, and check examples against the concept being taught. For workplace material, consider confidentiality, retention, provider access, and what information the task really needs. A fluent citation is not proof that the source exists or supports the claim.

Human review helps when the reviewer has time, evidence, and authority to intervene. It is not a universal safeguard by itself. Consequential systems also need enforceable limits and defined failure behavior.

Map the product around the model

An API credential used to contact a model provider is not necessarily the identity used by downstream tools. The Tool runner has a process security context and may use additional service credentials or delegated authorization. Each boundary needs its own checks.

Prompt injection and ordinary application flaws

Prompt injection attempts to make untrusted material redirect behavior. A retrieved document can provide information without permission to alter policy or authorize actions. Instruction hierarchy and model training can help, but independent application controls must govern sensitive operations.

Retrieval also requires ordinary access control. An index should not reveal another organization’s documents just because a model can search it. Generated output passed to a database, browser, or other interpreter needs context-appropriate protection. Database parameterization and browser output encoding have different purposes.

A refusal in chat is not evidence that every tool operation is blocked. Conversely, an unusual answer does not prove an unauthorized action occurred. Record the actual output, proposed operation, authorization decision, and observed effect separately.

Data, evasion, and privacy

Training-data poisoning attempts to influence learned behavior through manipulated training material. Changes to a retrieval corpus affect the application’s evidence without necessarily modifying model parameters. Both involve data integrity, but the mechanisms and validation differ.

Adversarial evasion concerns inputs crafted to induce an unwanted model outcome. The attacker constraints and required preservation of meaning or function depend on the task; there is no universal “change a few pixels” definition.

Membership inference estimates whether a record appeared in training. Data extraction seeks protected information. Model extraction seeks to reproduce or recover aspects of a model. These are different claims, and evidence for one does not automatically demonstrate another.

Explore prompt injection, data and output, and evasion and privacy for individual concepts.

Actions, text, and resource limits

Tool-enabled assistants need authorization tied to the actual person or workload, action, resource, and context. Narrow tool capabilities, isolation, destination restrictions, and suitable approval for consequential actions can limit harm. Model wording does not grant permission.

Text-only output can also disclose information or influence harmful decisions. Assess both communication and action. Unbounded requests, retrieval, generation, or tool loops may degrade availability or create substantial costs; define and verify suitable usage and duration limits.

The assistant proposes a legitimate action for the wrong account. Is the action safe?

No. A valid operation still needs the right actor, resource, and purpose. The permission check should reject an unauthorized reservation or update even when the model explains it convincingly.

Turn observations into improvements

Use a fictional scenario or an explicitly authorized evaluation environment. A useful finding identifies the untrusted source, affected decision, demonstrated consequence, and correction, while stating uncertainty. ATLAS and NIST terminology help describe these relationships consistently.

Continue with AI defense for layered controls. Evaluate model behavior and application enforcement together, with enough evidence to explain what improved and what remains uncertain.

Worked review: one assistant, several different results

Consider a fictional library assistant. Its user may reserve items only for their own membership. Source selection, generated text and optional actions are recorded separately. The following supplied records describe a simulated evaluation, with no real members or external effects.

• A1: Caller M14 is authenticated. The model-provider credential belongs to the application. A separate runner process uses account LibraryWorker; downstream reservations use a service credential constrained by an independent member-policy check.
• A2: The model proposes a reservation for M27. The policy compares it with caller M14 and rejects it. The effect record confirms that no reservation was created.
• A3: The draft cites document R9, but the complete retrieval record contains only R1 and R2. No external verification of R9 is supplied.
• A4: The broker permits at most three dispatched operations for this evaluation. It dispatches three and rejects a fourth proposal before dispatch. Per-operation cost and concurrent demand were not evaluated.
• A5: A separate privacy report labels a synthetic record “likely present in training.” It supplies neither ground-truth membership nor recovered protected text.

PredictShould the review label the assistant either “safe” or “compromised” based only on A2’s rejected reservation?

Neither label describes the evidence well. The wrong-member proposal is a model/application behavior to improve; the independent check prevented the observed action. A3 still has an unsupported citation, and A4 demonstrates only one bounded resource control. Keep each conclusion tied to its own record.

Improve the boundary that produced each result

For A2, retain the independent check using trustworthy caller identity and the actual proposed resource. Review useful allowed reservations as well as rejected ones; blocking every request would protect one outcome while making the service unusable.

For A3, do not publish the draft as sourced merely because the citation looks plausible. Require a retrievable source supporting the claim, or mark the answer unsupported and route it for review. Missing retrieval evidence does not prove that R9 is nonexistent everywhere, but it cannot support this answer’s attribution.

A4 needs additional budgets appropriate to cost, duration and concurrent use. A5 needs a clearly stated evaluation objective and evidence appropriate to that objective. Neither a membership estimate nor absence of recovered text settles the broader privacy question.

Model deliverable: four columns for claim, evidence, limit and correction. This preserves the difference between a model suggestion, an application permission and a demonstrated effect.