Evaluation needs failure cases

One idea. One situation. One reasoned decision.

How it works

An AI evaluation should include realistic tasks, misleading inputs, uncertain evidence, and operations that should be denied. Measure useful successful behavior and harmful or incorrect outcomes separately. A good average score can hide failure on a rare but consequential case. Keep test data appropriate for its use and revisit coverage when tools, models, prompts, or retrieval sources change.

Ninety-nine percent can hide the wrong ten cases

A fictional assistant must answer permitted routine requests and refuse messages addressed outside the approved audience. For this release exercise, the team requires every supplied wrong-recipient case to be refused.

Worked decision: the aggregate result is 990/1,000, or 99%. The critical refusal result is 0/10, or 0%. The release requirement is not met, even though the headline score looks strong. The denominator matters: correct routine answers cannot compensate for failure of this explicitly required boundary.

Report results by consequential behavior, along with the tested model, configuration, tools, and source set. Investigate the failed category and evaluate the revised system against both that category and permitted work. A system that refuses everything may satisfy a denial check while providing no useful service.

These ten cases are evidence about this test set, not a measured real-world incident probability. They also do not exhaust every future variation. The review should state coverage limits and revisit evaluation when important dependencies or intended uses change.

The key distinction: Aggregate quality can hide important failure categories.