The Replication Crisis

What it is

For most of the twentieth century, a published, peer-reviewed finding was treated as settled enough to build on: cite it, teach it, put it in a book. Starting in earnest around 2011, researchers began systematically re-running well-known studies with larger samples and analysis plans registered in advance, before anyone could see the data, and a striking share of the findings that had built entire subfields on top of them did not come back.

The clearest evidence comes from social priming, the specific claim that a brief, incidental exposure to a word, image or object changes real-world behaviour later on, not just what a person notices or recalls, but what they do. Across a wave of large-sample replications of this literature, 94 per cent produced smaller effects than the originals, and only 17 per cent reached significance in the original direction. Perhaps the most telling number in the whole crisis is a different one: the strongest predictor of whether a replication succeeded was not sample size or study design. It was whether one of the original authors sat on the replication team. That is evidence of how much unconscious latitude, in when to stop collecting data, which participants to exclude, which of several measures to report, separates "my effect replicated" from "it didn't."

Several of the field's most-cited results are now known casualties, each checked against the primary replication literature rather than taken on the original's word. Williams and Bargh's 2008 finding that holding a warm cup makes strangers seem warmer did not reproduce: Chabris and colleagues replicated it with triple the original sample, double-blind, and found effects close to zero, r = −.03 and .02, and three further high-powered replications totalling 861 participants found nothing at all. Money priming, Vohs and colleagues' 2006 claim that reminders of cash make people less helpful and more self-sufficient, failed across four high-powered replications run at two to four times the original sample sizes. "Romantic red," Elliot and Niesta's 2008 finding that a red frame raises perceived attractiveness, returned effects of d = 0.09 for men and d = −0.09 for women in pre-registered replications, close enough to zero to be noise, and a nine-study meta-analysis covering 640 participants found no effect at all. Ego depletion, the idea that willpower is a finite resource drained by use, fared no better once checked properly: Carter and McCullough's bias-correction reanalysis of a widely cited 2010 meta-analysis found an effect indistinguishable from zero, and the field's own 2016 multi-lab, pre-registered replication came back at roughly d = 0.003.

None of this makes the whole field worthless, and the crisis is not evenly spread. A robust tier survived intact: the misinformation effect, social facilitation, diffusion of responsibility, the mere exposure effect and processing fluency all held up under the same scrutiny that sank the studies above, some of them because they predate the priming era entirely and were never built on the same shaky foundations. What separates the tiers is a pattern worth learning as a standing test, since it can be applied without reading a single replication paper: the failures cluster in studies where a brief, incidental exposure is claimed to produce a large behavioural effect on something unrelated. Big effect, tiny intervention, no plausible dose-response, that shape is the signature of a finding unlikely to survive being run again.

In effect

Research: physical warmth and interpersonal warmth (Williams and Bargh, 2008)

The original study reported that participants who briefly held a warm cup of coffee rated a stranger as having a "warmer" personality than participants who held an iced coffee, a single moment of physical sensation reshaping a social judgement. Chabris and colleagues could not reproduce it. Running the study with triple the original sample and a double-blind design, they found effects of r = −.03 and .02, statistically indistinguishable from nothing, and three further high-powered replications, 861 participants combined, found no effect either. The finding had appeared in textbooks and TED talks for a decade before the replications caught up with it.

Everyday life: money priming (Vohs et al., 2006)

The original claim was that merely being reminded of money, seeing images of currency, unscrambling money-related word lists, made people act more self-sufficient and less generous toward others, a finding that travelled well because it maps onto an ordinary experience: thinking about finances and feeling less inclined to help. Four high-powered replications, each run at two to four times the original sample sizes, found no such effect, and questionable research practices were later identified in the original studies. The lived intuition the finding traded on, that money on the mind changes how generous a person feels, turned out to rest on a result that was not there.

Commerce: ego depletion and the checkout aisle

Roy Baumeister and John Tierney's Willpower (2011) took the finite-resource model of self-control mainstream, glucose claim included, and it shaped a great deal of commercial thinking about decision fatigue: retail checkout aisles stocked with impulse items, sales tactics timed for when a customer's willpower is assumed to be spent, corporate wellness programmes built around managing a resource assumed to run out. Carter and McCullough's bias-correction reanalysis of the field's own 2010 meta-analysis found an effect statistically indistinguishable from zero, and a 2016 multi-lab, pre-registered replication effort returned an effect of roughly d = 0.003, close enough to nothing to settle the question. The commercial architecture built on the willpower-depletion model was designed around a mechanism that, checked properly, is not there.

What it does not say

The crisis does not say psychology is unreliable as a field, only that it split into two tiers around 2011 and that almost nobody applying it commercially can currently tell which tier a given finding sits in. The robust tier, the misinformation effect, social facilitation, diffusion of responsibility, the mere exposure effect and processing fluency among them, is genuinely robust, older or better-tested than the casualties above, and the sorting rule exists precisely so the whole field doesn't get discarded along with its worst subfield. A failed replication is not a finding about the original researchers: the strongest predictor of a successful replication being an original author's presence on the replication team points toward unconscious researcher latitude. Finally, "94 per cent smaller, 17 per cent significant" describes social priming specifically; applying that exact figure to psychology findings outside that literature would repeat the mistake the sorting rule exists to prevent.

Who developed it

The replication crisis has no single originator. It surfaced through the cumulative, largely independent work of researchers across psychology, medicine and statistics through the 2010s who began pre-registering large-sample replications of well-known findings, rather than through one paper or one person.

---