Specification Curve Analysis

What it is

Start with the problem it was built for, which the people who use it state in their own words. Large social surveys carry hundreds of variables and tens of thousands of respondents, and their "many variables and observations are often explored with an analytical flexibility that marks small effects as statistically significant, thereby leading to potential false positives and conflicting results".

Unpack that. A researcher asking whether screen use goes with lower adolescent well-being has to make a series of ordinary decisions before any analysis exists. Which well-being measure, when the survey carries four. Which definition of the exposure, when the questionnaire asks separately about television, games, social media and total hours. Which covariates to control for, when sex, age, household income, parental education and ethnicity are all available and all arguable. Whether to treat hours as a continuous variable or to split respondents into heavy and light users. Four plausible outcome scales, three plausible control sets and two plausible exposure definitions produce twenty-four analyses, and conventional practice is to publish one of them.

Nothing about that is fraud. Each of the twenty-four is defensible, and the researcher has to pick. The difficulty is that the researcher picks after seeing the data, which means the published estimate is drawn from a distribution the reader never sees. Where effects are small, as they are throughout this literature, the spread across those twenty-four analyses can easily be wider than the effect itself, so the choice of specification determines the headline.

Specification curve analysis answers this by refusing to pick. The analyst enumerates every specification they consider defensible, runs all of them, and plots the resulting estimates in order of size. The output is not a number but a shape: a curve showing how many specifications produced a large effect, how many produced nothing, how many crossed zero, and whether the significant ones cluster around any particular analytic choice. A reader can see the full menu, and a claim built on one dish from it becomes visibly a selection.

That is a genuine improvement, and it is why the method spread quickly through psychology after the replication crisis made analytic flexibility a central worry. It converts a hidden decision into a published object.

And here is the limit, which this publication thinks is the single most useful thing to understand about the method. The set of specifications is itself a choice. Somebody decides which analyses count as defensible, and that decision sits upstream of everything the curve can show. Include total screen time and exclude social media considered separately, and the curve is a curve about screen time. Combine the sexes and the curve is about adolescents in aggregate. Control for a variable that lies on the causal path and every specification in the curve is deflated by the same amount.

So the method is honest about one kind of freedom and silent about another. It removes the freedom to pick a result from the curve. It does not remove the freedom to pick the universe from which the curve is drawn. A reader who treats a specification curve as a guarantee against analytic choice has mistaken a map of the choices for their absence.

In effect

Research: the same method, the same data, two headline numbers

This is the worked example that makes the limit concrete, and it is unusual in that both runs are published and both are correct.

The first run. Amy Orben and Andrew Przybylski, Nature Human Behaviour, 2019, pooled three large datasets, 355,358 adolescents in total: two American surveys and the British Millennium Cohort Study. Their result, in their own words: "The association we find between digital technology use and adolescent well-being is negative but small, explaining at most 0.4% of the variation in well-being. Taking the broader context of the data into account suggests that these effects are too small to warrant policy change." In plainer terms: out of everything that makes one teenager's reported well-being differ from another's, at most four parts in a thousand goes with digital technology use. Orben is first author and corresponding author, a fact worth stating because the work is frequently mis-credited to her co-author.

The second run, on the same three datasets. Jean Twenge, Jonathan Haidt, Jimmy Lozano and Kevin Cummins, Acta Psychologica, 2022, re-ran the method under four altered constraints: social media separated out from lumped total screen time, boys and girls analysed separately, potential mediators excluded from the control set, and scales weighted equally. Under the original configuration they reproduced the original result, median coefficients of about minus 0.01 to minus 0.04. Under the revised constraints, among girls, median coefficients ran from about minus 0.11 to minus 0.24. They report those associations as stronger than the ones for binge drinking, sexual assault, obesity and hard drug use, and conclude they "may have substantial practical significance".

One correction belongs here, and it is this publication's own. The Anxious Generation reports the 2022 paper's result in its endnotes as a single correlation coefficient. The published paper contains no such figure; it reports median coefficients across specifications, for girls, in the range above. Cite the range.

What the pair establishes is a finding about the literature rather than about children. Both numbers come from the same data by the same method, and the gap between them is produced by four analytic choices, each defensible and each contested. So the correct sentence is neither "the effect is tiny" nor "the effect is substantial". It is that the size of this association depends on how the data are cut, by more than the association itself.

Everyday life: the household budget that proves whatever you want

Anyone who has argued about money at a kitchen table has run an informal specification curve and reported one point from it. Did spending go up this year? It depends whether the annual insurance payment counts in the month it was paid or spread across twelve, whether the holiday is discretionary or a fixed commitment, whether to compare to last year or to the three-year average, and whether to adjust for the pay rise. Each choice is arguable. Each produces a different answer, and the person who runs the calculation picks the version that matches what they already thought.

The disciplined version is to run all of them and look at the spread, which usually reveals that the answer is "between a small rise and a small fall, depending", and that the argument was never about arithmetic.

Commerce and product design: the dashboard with too many denominators

A growth team reports that a redesign lifted conversion. Conversion by what definition: sessions, unique visitors, or signed-in users? Over what window: seven days, twenty-eight, or to the end of the quarter? Including or excluding the traffic from the launch campaign? Weighted by device, or pooled?

Those are specification choices, there are often dozens of defensible combinations, and in most organisations the one that gets into the deck is the one that moved. The practice that fixes it is familiar to anyone who has worked near a serious experimentation platform: the metric definition is written down before the test runs, and the platform reports every pre-registered cut rather than the best one. That is specification curve analysis arriving in commerce under a different name, and the reason firms adopt it is not scientific virtue. It is that a company which only ever sees the flattering cut eventually ships a year of changes that did nothing.

What it does not say

The method does not establish causation and makes no attempt to. Every specification in both curves discussed above is correlational, and a thousand correlational specifications are still correlational. Only designs that fix the order of events can license a causal claim, which is why the quasi-experiment page exists.

It does not guarantee a true answer, because the curve inherits every assumption shared by all its specifications. If a variable on the causal path is in every control set, every point on the curve is biased the same way, and the curve looks beautifully stable while being wrong. This is the technical charge, over-adjustment, that the 2020 exchange in Nature Human Behaviour turns on.

It does not make the people running it interchangeable. Two competent teams produced different headline numbers from the same data, and the difference was defensible judgement rather than error on either side.

And it does not settle the underlying argument. On the verified record, what is disputed in this literature is the size of the association and the specification that produces it, not its direction, which both camps agree is negative, and not whether adolescent mental health has deteriorated, which neither disputes. That exchange ran as a published comment and a published reply in the same journal, under everyone's names, with the reply's code and data deposited publicly. Both items are Matters Arising, an article type that carries no abstract; both bodies are paywalled and were not read by this publication, so what is said here about their contents comes from verified titles, figure captions, contribution statements and reference lists, and is identified as such. A widely repeated claim that the reply re-ran the curve and still found about 0.3 per cent of variance could not be verified and is not printed here.

Who developed it

The method is credited to Simonsohn, who is cited for it by name in the reference list of the 2020 reply, which is where this publication's attribution comes from. No year, co-authors or publication venue were verified at source on this pass, so no fuller citation is given, and the vault records the origin year as unknown rather than guessing one.

---