Randomised Controlled Trial

What it is

The central device is allocation by chance. If people are sorted by a coin, a table of random numbers or a line of code, then age, health, motivation and every other trait that might influence the outcome are spread across the groups by luck alone. The comparison between groups is then a comparison of the intervention. Observational comparisons lack this protection, because the people who choose or receive a treatment tend to differ from those who do not.

Ronald Fisher's The Design of Experiments (1935) is the text most closely tied to the statistical side of the idea. Its best-known illustration is a tea-tasting test: a woman claims to tell whether milk or tea was poured into the cup first, and Fisher proposes serving her eight cups, four of each kind, in random order. Guessing all eight correctly by chance has a probability of 1 in 70, about 1.4 per cent. The example shows how randomisation lets the experimenter state in advance what chance alone would produce.

In medicine the landmark is the Medical Research Council's trial of streptomycin in pulmonary tuberculosis, reported in 1948. John Crofton's 2004 account, in the James Lind Library, describes patients aged 15 to 30 with acute progressive bilateral pulmonary tuberculosis. Each sex at each centre had a numbered series of sealed envelopes, ordered by random numbers, each holding a card marked S for streptomycin or C for control. The next envelope was opened when a patient was approved for the trial, so the doctor could not choose who received the drug.

The vault's source for the modern commercial form is Adam Kucharski's Proof (2025). Kucharski's chapter on statistical testing follows the tools used to separate cause from coincidence, including Gosset's small-sample statistics, Fisher's null hypothesis tests and the randomised trial. He argues that the evidence hierarchy is a guide and not an oracle: a trial estimates an average effect, it cannot always be run, and its findings do not always transfer. The online version is the AB test, described on this site under AB Testing.

In effect

The streptomycin trial

Crofton reports that 55 patients received streptomycin with bed rest and 52 received bed rest alone. At six months there were 4 deaths among the 55 treated patients and 15 among the 52 controls. Three specialists graded monthly chest X-rays without knowing which patient had received which treatment, and the bacteriologists reporting sputum results were also unaware of allocation. Crofton credits Marc Daniels with coordinating the trial and names Daniels, Philip D'Arcy Hart and Austin Bradford Hill as responsible for its design, coordination, analysis and reporting. The trial ran for 15 months.

A coin flip in a bookshop

McGrath, Aronow and Shotwell (2016) ran a randomised trial of chocolate scent in an independent Canadian bookshop with an adjoining cafe. The trial lasted 31 days and the scent condition for each day was assigned by a coin flip. The scent was diffused out of sight, and staff and customers did not know which condition applied. It tested a claim that had been widely reported from an earlier study, and it shows that a randomised trial can be small and cheap when the unit of randomisation is a day.

Fifteen advertising experiments as a benchmark

Gordon, Zettelmeyer, Bhargava and Chapsky (2019) used 15 randomised advertising experiments at Facebook, covering about 500 million user-experiment observations, as the standard against which observational methods were judged. The methods, applied to the same data, often failed to recover what the experiments found. They usually overstated the effect of the ads and sometimes understated it. The experiment served as the reference because its allocation was random and the observational data were not.

What it does not say

A randomised trial does not say that the effect holds for every individual. It estimates an average across the people enrolled, a point Kucharski makes in his chapter on testing. Some participants may benefit, some may be unaffected and some may be harmed, and the average conceals which.

It does not say the result carries over. A trial on young adults with a particular form of tuberculosis supports a claim about similar patients under similar conditions. Transferring it to other groups needs further evidence or a judgement that the groups are alike in the ways that matter.

It does not say that a trial can always be run. Kucharski notes that randomised trials cannot always be run, and that natural experiments are used to fill some gaps.

It does not rescue a poor choice of outcome. An AB test is a randomised trial, and the Bing bug experiment reported by Kohavi and colleagues (2012) produced a correct, randomised measurement of increases in distinct queries and revenue that the authors explain as short-term and harmful to long-term value. Randomisation protects the comparison between groups and leaves the choice of measure untouched.

It does not guarantee precision. Lewis and Rao (2015) ran 25 randomised field experiments on advertising and found that the median confidence interval on return on investment was more than 100 percentage points wide. The design was sound. The sample needed to narrow the interval was, for most firms, out of reach.

Who developed it

The vault's note credits Austin Bradford Hill and the streptomycin trial as the origin of the modern medical form. Crofton's account names Hill as one of three people responsible for the trial's design, coordination, analysis and reporting, alongside Marc Daniels and Philip D'Arcy Hart. The statistical grounding in random allocation and significance testing is associated with Ronald Fisher. Online practice, in the form of the AB test, was formalised by Ron Kohavi and colleagues. Adam Kucharski is the vault's source for the historical account.

---