Security in depth · Unit 27 · Lesson 3 of 11

AI reliability and privacy risks

Separate misleading a prediction from learning about its data, and understand what evidence supports each claim.

7 minready

MITRE ATLAS

Helpful before thisAI data, retrieval, and safe output

After this lesson you can

Distinguish prediction errors, evasion, membership inference, and data extraction.
Explain why a confident output is not proof of a particular training record.
Match reliability and privacy controls to the system’s actual data flow.

A model can be unreliable without leaking private data, and it can leak information while making an accurate prediction. Those are different properties to evaluate.

Imagine a fictional photo classifier and a customer-support assistant. For the classifier, you might ask whether harmless changes in lighting make its predictions unstable. For the assistant, you might ask whether it exposes a person’s private information. The tests, evidence, and remedies differ.

A wrong prediction, a membership estimate, and reproduced source text are different observations. They require different evidence and controls.

Reliability has a threat model

An ordinary error is an incorrect result. Evasion is deliberate input manipulation intended to cause an unwanted decision during use. The evaluation must state what can change, what access the evaluator has, and what result would count as failure.

A harmless photo transformation and an arbitrary replacement image are different conditions. Likewise, a controlled benchmark result is not a universal statement that every deployed model fails. Record the model version, task, input population, baseline performance, and limits.

A classifier’s score can support a decision, but it does not establish a user’s permission. Keep identity and resource authorization independent of whether a content model gives something a favorable label.

Three privacy questions

Membership inference: was this particular record probably in the training data? That estimate can itself be sensitive, for example, membership in a health-related dataset. It is not proof from a single confident answer.

Training-data extraction: can output reproduce material attributable to the training data? Reproduced text may be public or sensitive. A privacy conclusion must identify what was exposed and why that exposure matters.

Inversion or reconstruction: what information about an input, attribute, or class can be inferred from model behavior? A reconstructed estimate is not automatically an exact recovered personal record.

These categories overlap in some research settings, but they should not be used interchangeably in a report.

Trace the source before naming the cause

Suppose a support assistant quotes a private ticket. The text might have come from retrieved context, the current conversation, a cache, or model parameters. Seeing the quote alone does not identify which source supplied it.

Review the application’s data flow first. A retrieval permission error requires a different correction from inappropriate training data. Replacing a model will not fix a shared cache that still returns another customer’s records.

Use fictional or appropriately controlled evaluation data. Record expected answers and evidence without collecting private material merely to increase the amount of proof.

Confidence needs a comparison

A membership inference claim needs a baseline, representative records, and measured mistakes. How often does the method incorrectly label an unseen record as a member? How does it perform on a comparable population? A selected success can hide many false conclusions.

Different tasks and models expose different information. Do not turn one research result into a prediction about every product.

Controls have specific coverage

Reduce unnecessary personal data in training and runtime context. Restrict dataset access, document retention, and review caches and traces. RAG can make source updates easier, but it does not automatically solve privacy or authorization.

Differential privacy can offer a defined mathematical guarantee about individual contributions. Its privacy unit, parameters, assumptions, and implementation matter. It does not secure an unrelated log store.

A useful conclusion names the property evaluated, the evidence, the uncertainty, and the control that addresses that exact exposure.

Terms you met

evasion membership inference differential privacy

Check yourself

No timer. No penalties. Read the explanation and try again whenever you like.

What does membership inference try to estimate?

Show the answer
Correct answer: Whether a particular record appeared in training data. It concerns training-set membership, and its conclusions require an evaluation with uncertainty and error rates.
A model is very confident about one record. Does this prove training membership?

Show the answer
Correct answer: No; confidence alone is insufficient evidence. A model can be confident about unseen records. A claim needs a suitable baseline and measured errors.
An assistant quotes a document supplied by its retriever. What is established?

Show the answer
Correct answer: The response used runtime context; training memorization is not established. Trace the data path before attributing the output to model weights.
What does a differential privacy claim need?

Show the answer
Correct answer: A defined privacy unit, parameters, assumptions, and correct implementation. A mathematical guarantee has a scope; the label alone is not a complete privacy assessment.

Try it

WriteA fictional classifier labels a known training example with high confidence. A separate support assistant quotes an internal document it retrieved. Write why neither observation alone proves training-data extraction, and identify one useful source of evidence for each system.

References