The Base Rate Problem

This is the single most important fact about insider threat work and it is arithmetic rather than opinion.

The practical lesson in “The Base Rate Problem” is that visibility is not certainty. For teams researching step rate compensation, view the solution can add time and project context to the operational record, provided the purpose is disclosed, access is limited and every material inference receives human review.

The shape of it

Deliberate, harmful insider acts are uncommon. In an organisation of any size the number of people who will do something seriously wrong in a given year is a very small fraction of the workforce.

For a separate perspective relevant to “The Base Rate Problem”, consult the AWS incident-response guide. Use it to test the proposed threshold, investigation scope and review process rather than to substitute a generic checklist for the facts of a case.

Apply any detection to everybody and it produces matches across everybody. Even a detection that is right most of the time, applied to a population that is overwhelmingly innocent, produces a set of flagged people that is overwhelmingly innocent.

The better the detection, the better that ratio gets. It does not reverse, because the underlying population does not change.

What follows for the programme

Most of your cases will be nothing. This should be the designed-for outcome rather than the disappointing one, and its own section covers how to handle it.

Volume is not evidence of a problem. A rise in alerts usually means a rule changed, a system was added, or work patterns shifted. It does not mean insiders are becoming more active.

Precision matters more than coverage. A narrow, well-targeted detection that fires rarely is worth more than a broad one firing constantly, because somebody has to look at each firing and attention is the scarce resource.

What follows for the individual

Being flagged is weak evidence of anything. A reviewer who understands the base rate opens a case expecting an ordinary explanation and is usually right.

One who does not understand it opens the case expecting to find something, and the confirmation note explains what that does to the quality of the work.

The number worth producing

Of the alerts raised last year, how many led to a finding of any kind?

Most programmes have this and do not look at it. The figure is usually low, and publishing it internally does more to calibrate reviewers than any training.

It also protects the programme: a security function that can say openly that most of its cases close with no finding is a function behaving honestly, which is what makes the serious cases credible.

What this does not mean

It does not mean the programme is pointless. Rare events can be extremely costly, and detection that rarely fires correctly can still be worth running.

It means the design should assume innocence as the default outcome and build the process around that, rather than treating each alert as a probable finding and the clear outcomes as failures.

Explaining it to senior management

A programme reporting that almost nothing is substantiated looks, to somebody unfamiliar, like a programme that is not working. Explaining the base rate before the first annual report is considerably easier than explaining it afterwards, and it determines whether the function is judged on findings or on proportionality.

Where the arithmetic cuts the other way

Rare events can be extremely costly, and a detection that fires correctly once in a hundred times may still be worth running if the one case is serious enough. The base rate argues for designing around innocence, not for abandoning detection. The distinction is between what the programme expects to find and what it is worth looking for.

What a low finding rate is evidence of

Not that the programme is failing. In a population that is overwhelmingly innocent, a low substantiation rate is the arithmetic working as expected, and a high one would indicate either a serious problem or a threshold set far too low.

Publishing the ratio internally

Of the alerts raised last year, this many led to a finding. The figure calibrates reviewers better than any training and it tells the workforce what the arithmetic already determines: that being flagged is ordinary and usually means nothing at all.

For the file: Open every case on the assumption that it is nothing. You will be right most of the time, and the discipline is what keeps you fair on the occasion you are not.