Measuring a Programme Honestly

Insider programmes report activity: alerts handled, cases closed, coverage extended. None of that says whether the thing is working or what it is costing.

The programme test in “Measuring a Programme Honestly” is whether a record leads to a better, explainable decision. Organisations considering the provider's detailed page for 7 minute rule payroll can use time and project evidence to locate operational questions, while governance, direct conversation and periodic review remain necessary to interpret any pattern responsibly.

The six

Alerts raised, by rule, so that the noisy rules are visible.

For a separate perspective relevant to “Measuring a Programme Honestly”, consult the BleepingComputer security coverage. Use it to test the proposed threshold, investigation scope and review process rather than to substitute a generic checklist for the facts of a case.

Cases opened, as a proportion of alerts. A programme opening a case for most alerts has no effective threshold.

Substantiated findings, as a proportion of cases opened. Low is expected; the base rate makes it inevitable.

Median time from opening to closure, and the longest.

People suspended who were subsequently cleared, which is the direct measure of harm to innocent people.

And process defects identified, which is the programme's contribution beyond catching people.

Why the fifth is the important one

It is the only figure that measures what the programme does to people who did nothing, and it is the one no standard reporting framework includes.

A programme suspending several people a year who turn out to be innocent has a proportionality problem regardless of how good its findings are.

Putting that number in front of oversight changes behaviour faster than any policy, because nobody wants to report it twice.

Why the sixth is the useful one

It is the output the rest of the organisation values and the one that justifies the programme in a budget conversation.

A security function that fixed eleven broken processes last year has a different standing from one that investigated forty people and found two.

Most programmes produce the first and never count it.

What not to measure

Detections as a success metric, which rewards noisy rules.

Coverage, which rewards monitoring more things regardless of value.

Cases opened, which rewards opening cases.

Each of these is in common use and each pushes the programme in a direction the next note describes.

The comparison worth making

Findings against harm: substantiated cases on one side, cleared suspensions and long clear cases on the other.

Stated plainly, once a quarter, to somebody outside the function.

A programme unwilling to produce that comparison is telling you something about what the comparison would show.

Where the numbers go

To an oversight function that is not the programme itself, which the audit note covers.

And in summary to the workforce, which sounds exposing and which the transparency note argues is the thing that makes the programme tolerable.

Reporting the harm alongside the findings

Substantiated cases on one side, cleared suspensions and long clear cases on the other, in the same document. A programme unwilling to produce that comparison has told you what it would show, and producing it quarterly is the single strongest discipline available.

The number the organisation actually values

Process defects identified and fixed. A security function that repaired eleven broken processes last year occupies a different position in the organisation from one that examined forty people and found two, and the first figure is usually available and uncounted.

The figure with no incentive behind it

People suspended and subsequently cleared. The programme has every reason not to produce it, which is precisely why oversight should require it and why it is the most informative number available.

Where the numbers go in practice

To oversight that is not the programme, and in summary to the workforce. Figures produced only for internal consumption are figures the programme can interpret to itself, which is not a check on anything.

Six numbers, quarterly

Alerts by rule, cases opened, findings, duration, cleared suspensions, process defects fixed. Together they describe what the programme is actually doing, and most programmes report none of them.

For the file: If you report one number upward, report people suspended and subsequently cleared. It is the figure that keeps the programme honest and the only one it has no incentive to produce.