Quasi-Experiment

What it is

The limit of a correlation is not that it is weak evidence. It is that it is the wrong kind of evidence. However large the sample and however careful the statistics, a cross-sectional association between two things cannot tell you which came first, and therefore cannot tell you which caused which. A teenager who is anxious may use a phone more; a teenager who uses a phone more may become anxious; a third thing may be driving both. The association is identical under all three stories.

A randomised controlled trial breaks the tie by deciding exposure with a coin. Random assignment makes the two groups comparable on everything, including things nobody thought to measure, so a difference afterwards can be attributed to the thing that was assigned. That is why trials sit where they do in the hierarchy of evidence.

For most interesting questions about societies, a trial is impossible, unethical or absurd. Nobody can randomly assign children to a decade of smartphone ownership. Nobody can randomise a country to receive broadband. The quasi-experiment is the next thing available, and the trick is to find a change in the world whose timing was decided by something unrelated to the outcome.

Two forms do most of the work in this literature. A difference-in-differences design compares the change in a place that got the thing against the change, over the same period, in a place that had not got it yet. It does not require the two places to be alike, only that they would have moved in parallel had nothing happened, which is an assumption and a checkable one. An instrumental variables design finds something that shifts the exposure and has no other route to the outcome, and uses only the part of the exposure that moves with it. A hilly region that is expensive to cable, a telephone network whose old architecture determined where broadband arrived first: these are instruments, and their credibility is entirely the credibility of the claim that they affect the outcome only through the exposure.

Both designs are real causal inference, and the good ones in this field are genuinely good. Which brings us to the thing this page exists to say.

A quasi-experiment licenses a causal claim about the population it was run on, the exposure it measured and the outcome it counted, and about nothing else. Those three constraints are not caveats at the bottom of a paper. They are the content of what was established. A beautifully identified effect of Facebook on undergraduates in 2005 is a beautifully identified effect of Facebook on undergraduates in 2005.

The error this guards against happens downstream, not in the papers. A study's design strength travels in summary and its sample does not. Writing picks up "a quasi-experimental study found that social media harms mental health" and drops the three constraints, so the authority of the design gets borrowed for a claim the design never tested. A good design on the wrong population is not weak evidence for the question you care about. It is strong evidence for a different question. Distinguishing those two sentences is most of what reading this literature well consists of.

In effect

Research: the staggered rollout, which is the textbook case

Luca Braghieri, Ro'ee Levy and Alexey Makarin, American Economic Review, 2022, used the staggered introduction of Facebook across United States colleges, coupled with student mental-health survey data from the same years, in a generalised difference-in-differences. Their finding, verbatim: "We find that the rollout of Facebook at a college had a negative impact on student mental health. It also increased the likelihood with which students reported experiencing impairments to academic performance due to poor mental health. Additional evidence on mechanisms suggests the results are due to Facebook fostering unfavorable social comparisons."

Why the design works is worth spelling out, because it is the clearest instance available. Facebook expanded college by college according to the company's own plan, not according to anything about the colleges' mental health, so the order of arrival functions as something close to an assignment. Students at a college that had just received it can be compared against students at a college that would receive it a few months later, and the comparison is between groups that were on the same trajectory.

Now the three constraints. The population is undergraduates. The exposure is Facebook between 2004 and 2006, which is before the smartphone, before the front-facing camera, before the algorithmic feed and before Instagram existed. The outcome is a student mental-health survey. It is the strongest quasi-experimental support the harm case has, and the technology the modern argument is about had not been built yet.

Research: the infrastructure build-out, with the author's own hedge

Marta Golin, Health Economics, 2022, combined confidential respondent coordinates from the German Socio-Economic Panel with technological features of the German telephone network, using the latter as an instrument for broadband access. Her result: "The results are suggestive that broadband Internet leads to worse mental health for women (primarily those aged 17-30) but not for men, thus widening the gender gap in mental disorders." Mechanisms named are worsened socialising behaviour and a reduced ability to cope with emotional problems.

Suggestive is her word and it should travel with the study. And note what the exposure is: a telecommunications infrastructure. Not a smartphone, not a social media account, not a feed. Between "broadband arrived in this postcode" and "this teenager's relationship with Instagram" there are several steps, each of which is an assumption rather than a measurement.

Everyday life: the roadworks that told you what the shortcut was worth

Two neighbours drive the same commute. The council closes one of the two routes for a month, for reasons that have nothing to do with traffic patterns, and one neighbour is forced onto the other road while the other is not. Comparing how their journey times change over that month, against each other, is a difference-in-differences, and it is more informative than any amount of asking people which route they prefer. Preference is correlated with everything about the person. The closure was correlated with nothing.

The reason it works is also the reason it is fragile: if the council chose that month because traffic was unusually heavy, the instrument is contaminated and the comparison collapses. Everything turns on why the change happened.

Commerce and product design: the phased rollout as an accidental experiment

Any company that ships a feature to one market before another has created a quasi-experiment, and most of them throw it away. A payments change that launched in Canada in March and in Australia in June supports a difference-in-differences on checkout completion, using Australia's pre-June trend as the counterfactual for Canada. It is nearly free, the data already exists, and it answers a question that a before-and-after comparison in Canada alone cannot, because something else happened in March too.

The school phone-ban literature contains two commercial-adjacent instances. Sara Abrahamsson's event study used Norwegian middle schools' own decisions to ban smartphones. Hunt Allcott and five colleagues ran a staggered difference-in-differences across United States schools adopting lockable pouches, using among other inputs the sales records of the largest pouch provider, which is a named data partner and should be disclosed whenever the study is cited. That study is an NBER working paper and is not yet peer-reviewed, which must be said every time.

What it does not say

The designs on this page do not deliver a causal claim about anyone other than the people in them. This is the whole point of the page and it bears repeating in the negative: nothing here establishes that smartphone or social media use caused the deterioration in the mental health of children aged roughly nine to fifteen, because no design of this class was run on that population with that exposure.

Four designs in this literature can license a causal claim, and their populations are undergraduates in 2005, adult volunteers in 2018, German adults and young adults, and national hospital and administrative registers. Not one of them is a nine- to fifteen-year-old with a smartphone. The best causal evidence in the field concerns adjacent populations and adjacent exposures, and the inferential distance from there to a developmental claim about children is the thing most summaries leave out.

It is equally not a refutation. Absence of a demonstrated effect, in designs mostly not aimed at the right population, is not a demonstration of absence. A population-level change in the social environment of childhood is exactly the sort of thing that designs comparing individuals within a population are worst at detecting, and that is a real argument rather than a debating point.

Nothing on this page is a criticism of any of the studies named. Each states its own population plainly, and several state their own hedges. Golin wrote suggestive; Allcott and colleagues wrote suggestive evidence and appear to recover. The limitation lives in how the work is summarised, not in how it was done.

Two further quasi-experiments exist in this literature whose outcome is an adolescent clinical record rather than an adult self-report, a Spanish fibre-optic rollout study and an Italian broadband study. Both were verified at bibliographic level only by this publication and neither is cited here. One repository version of the Spanish abstract reports a finding on adolescent suicide and another does not, so given the subject matter it is not printed on anything short of the published paper. Those two papers are the biggest open gap in this evidence base.

Who developed it

No single originator. The quasi-experiment is a standing design family in econometrics and the social sciences rather than one person's proposal, and this publication verified no attribution for the term, so none is given. The vault's note records the origin as unknown rather than naming a plausible candidate.

---