Survivorship Bias: The Bullet Holes That Weren't There

During the Second World War, the United States military examined bombers returning from missions over Europe and mapped where they had been hit. The pattern was clear and consistent: dense clusters of bullet holes along the wings, the fuselage, and the tail gunner's position, with the engines and cockpit comparatively clean.

The obvious conclusion was to reinforce the areas taking the most fire. Armour is heavy, fuel is finite, and you cannot armour everything — so put it where the bullets are.

Abraham Wald, a statistician working with the Statistical Research Group, reached the opposite conclusion. Armour the engines, he said. The parts with no holes.

His reasoning inverted the entire dataset. The military was not looking at a map of where bombers got hit. It was looking at a map of where bombers could get hit and still fly home. The engines showed no damage in the sample not because they were rarely struck, but because aircraft struck there were lying at the bottom of the Channel, unavailable for inspection.

The data was not wrong. The data was the survivors, and every conclusion drawn from it inverted the truth.

The mechanism: a sample that filtered itself

Most reasoning errors are errors of analysis — you had the right data and drew the wrong inference. Survivorship bias is more insidious, because it corrupts the input. By the time you begin analysing, the failures have already been removed, silently, by the process itself.

This is what makes it so hard to catch. There is no obvious gap in the spreadsheet. Nothing looks missing, because the missing entries never generated a record. The sample feels complete precisely because you are looking at everything that remains.

The general structure recurs everywhere:

A process runs. Companies are founded, planes fly missions, funds are launched, treatments are attempted, buildings are constructed.

Failures exit the observable population. Firms dissolve and stop filing reports. Funds close and drop out of the performance index. Buildings are demolished. Aircraft do not return.

Someone studies what remains and draws conclusions about the process — conclusions that would be different, often reversed, if the exits were included.

The critical insight is that the trait shared by survivors is only informative if it is rarer among the failures. Without failure data, every shared characteristic of the successful looks causal. Including the characteristics that actively increased the chance of failing, which will be over-represented among survivors purely because the survivors are the lucky tail of a risky strategy.

Where it does real damage

Business advice is almost entirely constructed from survivors. Every study of "what successful companies have in common" examines firms that succeeded. Bold vision, aggressive risk-taking, refusing to pivot, betting the company on one product — these appear in the biographies of the triumphant. They also appear in the biographies of the thousands who did exactly the same and are not available for interview because the company no longer exists.

The famous "successful founders dropped out of university" observation is the cleanest case. To know whether dropping out helps, you would need the outcomes of everyone who dropped out — an enormous population, overwhelmingly not billionaires. The survivors are visible; the denominator is invisible; and the advice derived from the ratio is worse than useless because it actively recommends increased risk.

Investment track records are systematically inflated. Funds that perform badly close, and closed funds typically leave the indices used to compute historical returns. The reported average return of "funds in the index" is therefore the average of the ones that survived long enough to remain in it. The same mechanism makes almost any long-run performance comparison flatter than reality.

"Things were built better in the past." The buildings, furniture, and tools that survive from a century ago are the best-built examples; the shoddy majority decayed or was discarded. You are comparing today's full production run against history's curated top percentile. This is survivorship bias wearing the costume of nostalgia.

Medicine and science are affected structurally. Studies with striking results get published; studies finding nothing often do not. The published literature is therefore a survivor population, systematically over-representing positive findings — which is a substantial part of why so many published results fail to replicate.

Advice from the successful is doubly filtered. Not only do we hear from survivors, but survivors reconstruct their own histories as narratives of skill and judgement. Luck is genuinely difficult to perceive from the inside; a decision that worked feels, in retrospect, like a decision that was correct.

How to defend against it

Ask explicitly: where are the failures, and what would they say? This one question catches most instances. Before accepting that a trait causes success, ask what proportion of people with that trait failed. If the answer is unknown, the claim is unsupported — not wrong, but unsupported, which is a different and more honest position than either belief or dismissal.

Look for the base rate. Survivorship bias is fundamentally a missing-denominator problem, which connects it directly to the base rate fallacy. "Ten of the world's largest companies were founded in garages" means nothing without knowing how many companies were founded in garages in total.

Distrust reasoning from a handful of vivid cases. The more compelling and memorable the example, the more likely it reached you because it was exceptional — which is precisely the selection mechanism that makes it unrepresentative.

Deliberately study failures. This is the practical inversion, and it is why post-mortems, incident reports, and accident investigation add so much value: they reconstruct the invisible population on purpose. Aviation safety improved dramatically not by studying successful flights but by investigating every crash exhaustively. It is also an application of inversion — asking what guarantees failure rather than what produces success.

In power-law domains, expect this bias to be at its most severe. Where outcomes follow a power law, the survivors are not merely a bit better than average — they are extreme outliers, and generalising from them to a strategy is close to meaningless.

Wald's recommendation was adopted, and the reasoning behind it outlived the war by a considerable margin. His actual contribution was not an insight about armour. It was noticing that a complete-looking dataset had a hole in it exactly the shape of everything that had already been destroyed — and that the most important information in the room was the information that had not made it back.