Survivorship Bias in Success Stories
Success stories are chosen because the outcome was good, so whatever the winners share looks like a cause. Abraham Wald's returning bombers, a database of 2,071 US equity funds and a fighter pilot's survival figure show how the missing cases change the answer, and what it takes to put them back.
Answer to the question
Why do success stories mislead us?
They are chosen because the outcome was good, so any trait the winners share looks like a cause. The failures that shared it are not in the sample. In a study of 2,071 US equity funds, ignoring the funds that closed inflated average annual performance by up to about 1 per cent.
Key takeaways
- Abraham Wald's wartime memoranda, reprinted in 1980, estimated how hits fell on downed aircraft from data on returning aircraft alone; the popular armour-the-engines story is, in a historian's review, a plausible reconstruction.
- Mark Carhart and colleagues found the annual bias in average fund performance grew from 0.07 per cent in one-year samples to about 1 per cent in samples longer than 15 years, with about 35 per cent of 2,071 US equity funds gone by the end of 1995.
- Jerker Denrell (2003) showed that risky practices can look beneficial among survivors even when they make no difference across everyone who tried, because failures are undersampled.
- Hasard Lee's report that fighter pilots' survival climbs sharply after ten combat missions describes pilots who had already flown ten, and by itself cannot separate learning from the removal of the unlucky.
- A winner's account is good evidence that something can be done and how; it cannot say how often the same habit ends in nothing.
The best case for learning from winners is that they are the ones available to be questioned, and whoever built the company or flew the missions knows things a bystander does not. The case is sound as far as it goes. Survivorship bias begins at its edge, when every question is put to people who came through and the answers are read as though they described everyone who tried.
What the returning bombers could not show
During the Second World War the mathematician Abraham Wald worked at the Statistical Research Group at Columbia University on how many hits an aircraft could take. His work survives as eight memoranda, reprinted in 1980 by the Center for Naval Analyses. Five of the eight deal with the chance that a plane survives a given number of hits, and they rest on one fact about the data: it came only from planes that returned.
The planes that did not return could not be examined. Wald's tool was an equation that lets an analyst estimate how hits were distributed on the downed planes from the survivors alone.
The planes that did not return could not be examined.
The popular retelling has Wald telling the military to armour the engines, because the returning planes had no holes there. A historical review for the American Mathematical Society calls that version a plausible reconstruction, notes that the memoranda say nothing about what the military should do, and finds no sign that Wald ever drew the pictures that circulate under his name.
What the record supports is smaller and more useful. The problem was recognised in 1940s statistics and put into an equation. Survivorship bias is the name for what that equation corrects.
Closed funds change the average
Money is the cleanest place to watch survivorship bias work, because failure there is a deletion. Mark Carhart and colleagues studied 2,071 US diversified equity funds between 1962 and 1995. By the end of 1995, 1,346 were still operating, which leaves 725 gone, about 35 per cent of the sample.
Averaging only the funds that remained flatters the average, and the flattery grows with the length of the record. In their words, the annual bias "increases from 0.07% for 1-year samples to 1% for samples longer than 15 years." A fund that closes after a poor run takes its record with it. A ten-year chart of a fund family's current products is a chart of the products that were not closed.
Management advice has the same flaw built in. Organisations learn from the firms they can observe, and the firms they can observe are mostly survivors, so failures are undersampled. That is the argument of Jerker Denrell's 2003 paper in Organization Science, and his conclusion is that risky practices can look beneficial among the survivors even when they make no difference across the whole population.
Suppose bold strategies and cautious ones have the same average return, but the bold ones scatter widely. The best results then come disproportionately from the bold, and so do the worst. Only the best are invited to speak at conferences, and a study of them finds boldness.
The same logic reaches the shelf of habits-of-the-successful books. Their cases are chosen because they ended well, a close cousin of [judging a decision by how it turned out](https://thewhyweproject.com/articles/how-outcome-bias-judges-a-decision), and the rule for choosing them is rarely declared. Admiration for the winner adds a second distortion, the one behind [the halo effect](https://thewhyweproject.com/articles/how-far-the-halo-effect-spreads). Journals apply the same filter to findings, since the studies that reach print are survivors of a selection, a problem the [replication crisis](https://thewhyweproject.com/articles/the-psychology-you-learned-didnt-replicate) made expensive.
Counting the pilots who did not reach mission eleven
The seed for this piece is Hasard Lee's The Art of Clear Thinking (2023), a fighter pilot's guide to decisions. The book never mentions survivorship bias, so the connection below is ours, not Lee's. At pages 82 to 83 Lee reports that problems were acute before a pilot's first ten combat missions, after which survival rates climbed sharply, and he attributes the finding to Air Force studies of the Vietnam War. The book names no author, volume or date for them.
The nearest authoritative statement we found is in Air & Space Forces Magazine (2005), which says only that other studies of pilot losses "found that surviving the first 10 combat missions is critical." It gives no rate and does not tie the finding to a named report, so we could not trace the figure to its source.
Here survivorship bias runs the other way. In Wald's problem the evidence of where the danger lay was in the planes that did not come back. A survival rate computed for missions after the tenth describes pilots who had already flown ten, and it can rise without anyone having learned anything. If some pilots were far likelier to be shot down than others, the likeliest had mostly gone by mission eleven, and those remaining look safer because the unlucky and the unprepared were removed.
That is our reading of the logic, not a finding about the Air Force data behind the figure, which we have not seen. The figure as the book states it cannot separate learning from removal, and the book does not raise the question.
Both cases are repaired the same way. Wald rebuilt the missing planes from the hits on the planes present, and a fund study puts the closed funds back. The pilot figure needs the count of pilots at each mission, including those who did not reach the next.
Nothing above shows that the habits of successful people are useless, only that survivorship bias makes them hard to read. A winner's account is good evidence that a thing can be done and how it was done. A story of the early riser who built an empire is evidence that early rising and empires are compatible.
A story of the early riser who built an empire is evidence that early rising and empires are compatible.
What it cannot supply is how often the same habit ends in nothing. A fair test would sample entrants before the outcome is known, every firm founded in a given year or every pilot who began training, and then ask which habits predicted who was still standing. Carhart's team built that kind of sample for funds by keeping the closed ones in. For habits, nobody has counted the entrants who did the same and stopped, and until someone does the claim has no comparison.
Common questions
Why do success stories mislead us?
They are chosen because the outcome was good, so any trait they share looks like a cause. Denrell showed in 2003 that risky practices can appear beneficial among survivors even when they make no difference across everyone who tried, because failures are undersampled.
What is survivorship bias in simple terms?
It is drawing conclusions from the cases that made it through a selection process while ignoring those that did not. Wald's returning bombers, a fund database that omits closed funds and a book of successful founders share the flaw, since the missing cases may differ from the visible ones in exactly the way that matters.
How big can survivorship bias be in investing?
In Carhart and colleagues' sample of US equity funds, the annual bias in average performance grew from 0.07 per cent in one-year samples to about 1 per cent in samples longer than 15 years. About 35 per cent of the funds had disappeared by the end of 1995.
Sources
Lee, H. (2023). The Art of Clear Thinking: A Fighter Pilot's Guide to Making Tough Decisions. London: Torva
Wald, A. (1980). 'A Method of Estimating Plane Vulnerability Based on Damage of Survivors.' Statistical Research Group memoranda, reprinted by the Center for Naval Analyses. https://cna.org/reports/1980/a-method-of-estimating-plane-vulnerability
Casselman, B. (2016). 'The Legend of Abraham Wald.' American Mathematical Society Feature Column, June 2016. https://www.ams.org/samplings/feature-column/fc-2016-06
Carhart, M. M., Carpenter, J. N., Lynch, A. W., & Musto, D. K. (2002). 'Mutual Fund Survivorship.' The Review of Financial Studies, 15(5), 1439-1463. doi:10.1093/rfs/15.5.1439
Denrell, J. (2003). 'Vicarious Learning, Undersampling of Failure, and the Myths of Management.' Organization Science, 14(3), 227-243. doi:10.1287/orsc.14.2.227.15164
Hebert, A. J. (2005). 'Red Flag With a Difference.' Air & Space Forces Magazine, August 2005. https://www.airandspaceforces.com/article/0805redflag/