Signal Detection Theory

What it is

Every yes-or-no decision under uncertainty has two independent components. Sensitivity, often written as the distance between two distributions, is the real ability to tell signal from noise. The criterion, or bias, is where the person draws their line.

Two people with identical ability will produce different hit rates if they set different thresholds, and one person's hit rate can rise with no change in ability at all. Reporting accuracy without separating the two therefore confuses skill with temperament.

Receiver operating characteristic analysis is the usual method: plot the hit rate against the false-alarm rate across thresholds, and the shape of the curve reports sensitivity independently of where any particular person drew their line.

In effect

The application this site came to it through is misinformation research. Evaluations of inoculation games compared average ratings of true and fake headlines before and after play, which cannot distinguish a genuine gain in discrimination from a general rise in scepticism. Reanalysed with the curve, the games moved the bias and not the sensitivity.

The same logic applies to anything scored by whether people correctly reject bad examples: fraud-awareness training, scam-spotting campaigns, advertising-claim checklists, and medical screening, which is where much of the method was developed.

Philip Higham's background in memory and metacognition is why this critique arrived from outside the misinformation literature rather than inside it, which is a pattern this site keeps finding.

What it does not say

It does not say which threshold is correct. Sometimes caution is the right response to a dangerous environment, and the framework is silent on what a person should want.

It does not make a weak study strong. Separating the terms can only report what the data contain; it cannot supply discrimination that was never measured.

And this record does not assign the framework an originator. It long predates the application discussed here and we have not checked its history.

---