The Metrics That Corrupt It

Measure a programme on activity and it produces activity. The particular activity it produces is enquiries into colleagues, which makes the choice of metric unusually consequential.

The programme test in “The Metrics That Corrupt It” is whether a record leads to a better, explainable decision. Organisations considering see the complete overview for fte meaning can use time and project evidence to locate operational questions, while governance, direct conversation and periodic review remain necessary to interpret any pattern responsibly.

Cases opened

Rewards opening cases. Produces a lower threshold, more subjects, more people put through a process.

For a separate perspective relevant to “The Metrics That Corrupt It”, consult the CERT-EU security guidance. Use it to test the proposed threshold, investigation scope and review process rather than to substitute a generic checklist for the facts of a case.

The threshold note argues for a stated bar; a target on case volume removes it in practice whatever the policy says.

Detections or alerts

Rewards noisy rules and broad monitoring. A programme measured on detections has no reason to retire a rule that fires constantly and finds nothing, and considerable reason to add more.

Which produces the volume problem, which produces the fatigue, which produces careless handling of innocent people.

Coverage

Rewards monitoring more systems and more people regardless of whether anything is learned.

Coverage is easy to report and feels like progress. It is the metric most likely to extend a programme into areas where the proportionality argument has never been made.

Time to close

Looks sensible and pushes toward quick conclusions, which in practice means conclusions reached before the ordinary explanation was found.

Speed matters — the timeframes note argues delay harms people — but as a reported target it rewards closing rather than resolving.

What these have in common

Each is easy to count, each describes effort rather than outcome, and each can be improved by doing more of something that costs other people.

That last property is the test. A metric that improves when the programme becomes more intrusive is a metric that will make it more intrusive.

The replacements

Findings against cases opened, which rewards precision rather than volume.

Cleared suspensions, which is a cost and should be reported as one.

Process defects fixed.

And median duration for cases that found nothing, which rewards resolving innocent cases quickly without rewarding premature closure of substantiated ones.

The target question

Ideally none of these carries a target at all. An insider programme with numeric targets on activity is being paid to find people, and that incentive is visible to everybody subject to it.

Where targets are unavoidable, put them on the harm measures rather than the activity ones: reduce cleared suspensions, reduce time to clear.

Those are targets a programme can only meet by being more careful.

Why targets are worse than measures

A measure that is watched informs; a measure that is targeted directs behaviour. An insider programme with numeric targets on activity is being paid to find people, and everybody subject to it understands that, which affects reporting long before it affects anything else.

The replacement set

Findings against cases opened, cleared suspensions, median time to clear, and process defects fixed. Each can only be improved by being more careful, which is the property the activity measures lack.

The test for any metric

Could this be improved by investigating somebody who did nothing wrong. If yes, it will be, and the improvement will be reported as progress.

Targets on harm rather than activity

Reduce cleared suspensions, reduce time to clear. These can only be met by being more careful, which is the property the activity measures lack entirely and the reason they corrupt.

Watched rather than targeted

A measure that is observed informs; one that carries a target directs behaviour. For a function with the power to examine anybody, that distinction decides how many people get examined.

For the file: Ask of every programme metric: could this be improved by investigating somebody who did nothing wrong? If yes, it is the wrong metric.