Case: a quieter queue, a missed change

Your job: review the review queue

You help the fictional Maple repair café manage booking permissions. Its reviewers want fewer unnecessary alerts, but they also want to find unapproved changes. Two proposed rules, A and B, have already been evaluated against the invented evidence below. Your task is to recommend a next step, not to configure a system.

The unit is one completed permission change. An independent review of change records and approvals labels twelve cases: four unapproved and eight approved. “Positive” means unapproved under this fictional policy; it does not prove malicious intent or identify a person. The reviewers checked cases with and without alerts, so the sample includes potential misses. Each change counts once, even if several logs describe it.

Stage 1: choose the right denominators

PredictBefore reading on, which denominator answers “How many of A’s alerts deserve investigation as unapproved changes”: four, six, or twelve?

Six alerts. Three are true positives and three false positives, giving precision 3/6 = 50%. Recall asks a different question: A detects three of the four actual unapproved changes, so recall is 3/4 = 75%. Its false-positive rate is 3/8 = 37.5%, using the eight actual approved changes.

The four confusion counts for A are TP 3, FP 3, FN 1, and TN 5. They sum to twelve. Google Developers’ classification reference explains these denominators; the café’s labels and evidence are original to this exercise. A percentage calculated correctly can still answer the wrong question if you change the population or label definition halfway through.

Stage 2: inspect the price of an exception

Rule B adds an exception for a familiar administrative workflow. It suppresses E and F, both approved changes, but also A, an unapproved change within that workflow. It still alerts on B, C, and G. Its counts become TP 2, FP 1, FN 2, and TN 7. Precision rises to 2/3, about 67%, while recall falls to 2/4, or 50%. Its false-positive rate falls to 1/8, or 12.5%.

Assume eight minutes of initial review per alert and a 32-minute budget for this batch. Rule A demands 48 minutes; B demands 24. These estimates exclude deeper investigation and assume no duplicate grouping. The smaller queue fits, but the newly missed case is a real cost. Approval of a workflow is not blanket approval of every change within it.

PredictShould the café adopt B because its precision is higher and its queue fits the budget?

Those are advantages, not a complete decision. B loses case A. The owner must weigh the consequences of missed changes against review capacity. Better approval context, defensible prioritization, additional capacity, or a narrower reviewed exception are alternatives worth testing. The table alone does not show which will work.

Stage 3: notice what the table cannot say

Four further changes occurred on a second service while its telemetry was unavailable. They have not been independently reviewed and are not among A-L. Do not append them as true negatives because there were no alerts, or as false negatives because collection failed. Their labels and the rules’ behavior with complete data are unknown.

OWASP’s logging guidance makes collection quality and context part of useful event records. NIST’s response guidance connects detection to analysis and action. Here that means assigning someone to assess the coverage gap and deciding how to report it, alongside reviewing the rules themselves. Neither metric is a measurement of coverage for every café service.

Model decision and handoff

“A detects more of the sample’s unapproved changes but exceeds the stated review budget by sixteen minutes. B fits the budget and improves precision, at the cost of an additional miss. We will ask the service owner to approve a temporary prioritization and resource decision, examine the exception that lost A, and evaluate a revised design on separately reviewed cases. Four unobserved changes remain an open coverage issue.”

Another defensible decision might accept B temporarily where the missed class has lower agreed consequences and another effective control covers it. That requires evidence, an owner, and a review date; none is supplied here. A tiny constructed sample teaches arithmetic and tradeoffs. It cannot establish future performance, statistical certainty, or the cause of any unapproved change.