Detection: why a good alert needs context

A smoke alarm that rings every time you make toast will eventually be ignored. Security alerts have the same human constraint: they need evidence, context, and a clear next step.

Precision: The fraction of positive alerts that are true positives.

Start with a question

An event records something that happened. A detection applies logic to events and context; an alert asks someone or something to act. Start with a hypothesis such as “an important permission changed outside its approved window.” Define the data required, expected benign explanations, and the action a responder can take.

Missing or delayed logs can make a rule silent. Silence is not proof that nothing happened. Monitor collection health and timestamps alongside the rule.

Rare events change the arithmetic

Precision is the fraction of alerts that are true positives. Recall is the fraction of actual positive cases the detection finds. False-positive rate uses a different denominator: the benign cases. A small false-positive rate can still produce many alerts when benign activity greatly outnumbers malicious activity.

The calculator below uses a fixed synthetic population, not measured product performance. It assumes ten malicious events among ten thousand, and detection of nine of them. Change only the false-positive rate and watch the alert workload.

Treat rules as maintained products

Test against representative synthetic or appropriately handled historical data, including ordinary activity and known positive cases. Record what the rule cannot see. Suppression may reduce noise while hiding real events; document that tradeoff.

Provide a playbook with evidence to inspect, escalation criteria, and a responsible owner. Track useful outcomes and analyst burden rather than celebrating the raw number of alerts. Feed reviewed results back into the rule and its data requirements.

EXPLORE THE CONCEPT

What would make this alert useful?

Compare three detection designs, then explore the arithmetic.

Every failed login triggers an incident

Failures have many ordinary causes. A context-free rule may create a large queue without identifying a meaningful pattern.

An unexpected privileged change includes its approval context

The analyst can compare the change with a known expectation. The result still requires investigation.

The rule is quiet while collection is broken

No data is a coverage failure, not reassuring evidence. Telemetry health needs its own monitoring.

A simplified learning model. It connects to no systems and uses no real data.

Work a rule from question to decision

Imagine a fictional community library with two booking administrators. Its question is: “Did someone give an account booking-administrator rights without approval?” That is more useful than “Find suspicious activity” because it names the action, protected capability, and expected evidence. An administrative event can show that a change occurred; a separate approval record helps judge whether it was expected. Neither source alone proves who was at the keyboard.

Write down the unit being counted: here, one completed permission change. Three log records describing that same change are not three independent cases. Define a positive label as a confirmed unapproved change after review. A permitted emergency change documented through an agreed emergency process belongs among the benign cases, even if its timing looks unusual. This label definition decides what your numbers mean.

The library reviews 100 fictional changes independently of the rule. Eight were unapproved. The rule flags six of those eight and twelve of the 92 approved changes. Its precision is 6/18, about 33%; its recall is 6/8, or 75%; its false-positive rate is 12/92, about 13%. These are results for this labeled sample. They are not guarantees about next month’s activity or services whose logs were missing.

PredictThe rule produces half as many alerts after a new exception. Has detection improved?

Not enough information. Check which alerts disappeared. Removing six approved cases reduces unnecessary work; removing three unapproved cases also loses coverage. Compare the same reviewed population and label definition, then examine future behavior separately.

Make the investigation affordable

Suppose each of the eighteen alerts takes ten minutes to check. That is three hours of triage, before any substantial incident response. If the on-call reviewer has one hour, assigning an owner alone has not made the design workable. Consider better approval context, grouping duplicate records, prioritizing consequential changes, or adding review capacity. Each choice needs a stated tradeoff; suppressing an entire administrator account could hide its unexpected activity.

An alert should carry enough context for its first decision: affected account and privilege, event and receipt times, approval reference if present, collection status, and the responsible service. Restrict sensitive detail to people who need it. OWASP’s logging guidance matters here because excessive personal data or credentials create another problem, while absent identifiers make correlation difficult.

Keep a small rule review record

Record the question, relevant services, data dependencies, label definition, sample dates, confusion counts, review cost, owner, and next review date. Include the two missed unapproved cases: what differed, whether the necessary evidence existed, and what further coverage would require. Also record which systems were excluded. “No alerts” on a system that stopped reporting is an unknown, not a successful negative test.

Revisit the rule when permission workflows, logging formats, or responsibilities change. NIST’s incident-response guidance places detection within a wider preparation and response process; a technically correct match still needs an operational decision. Practice a smaller comparison in the detection review case.

Turn the idea into a decision

Ask what an alert means, what evidence supports it, and who can act. Precision and recall describe different aspects of a detection.