Test-Retest Reliability

What it is

If an instrument measures a durable property of a person, such as a settled attitude, then measuring the same person twice should give similar results, apart from ordinary noise. Test-retest reliability counts that similarity. Everyone takes the instrument at two points, and the correlation between the first and second scores across the whole sample is the figure. A high correlation means people keep their rank order. A low one means rank order shuffles between occasions, so a single score is a poor guide to where a person will land later.

Reliability is a ceiling on validity. A score that wobbles for the same person cannot forecast an outcome as well as a steady one, however well the underlying idea is formed. That is why reliability and Predictive Validity are reported together, and why a reliability figure matters when a score is used on individuals.

It is worth separating two things that are easy to run together. Internal consistency asks whether the items inside one administration agree with each other. Test-retest reliability asks whether the whole result repeats over time. An instrument can do well on the first and poorly on the second.

In effect

A 2017 study in Personality and Social Psychology Bulletin separated those two ideas for implicit measures. Bertram Gawronski, Mike Morrison, Curtis Phills and Silvia Galdi asked whether individual differences on implicit measures stay more consistent over time than individual differences on explicit measures. A common assumption is that implicit measures, being harder to deliberate over, should resist change. The authors distinguished that claim from a second one, that implicit measures give more stable individual differences, and tested the second.

They ran two longitudinal studies across three domains: self-concept, racial attitudes and political attitudes. In both, implicit measures were significantly less stable over time than their conceptually matched explicit measures, with a weighted average correlation of about 0.54 against about 0.75. Internal consistency was comparable between the two kinds of measure, so the gap in stability is not explained by one being noisier within a single sitting.

The finding bears on how the Implicit Association Test is used. The authors of the 2015 defence of the test, Anthony Greenwald, Mahzarin Banaji and Brian Nosek, themselves state that its modest test-retest reliability makes it risky for classifying individuals. The 2017 figures describe implicit measures as a family and do not isolate the Implicit Association Test by name in the abstract read for this page.

What it does not say

A lower correlation across occasions does not show that the measure is worthless. Some of the movement between occasions may be real change in the person, and the studies do not separate genuine change from measurement noise on their own.

It does not show that an implicit measure is invalid, only that a single score is a weaker guide to a later score than a stated attitude is. For population-level uses, where scores are averaged over many people, a modest individual-level correlation is less damaging.

The 2017 figures are weighted averages across domains. They are not a reliability coefficient for any one instrument, and they should not be quoted as one.

---