Research library Selection bias and paradoxes
The Planes That Came Back.
In 1943 Abraham Wald worked out where bombers were most vulnerable using only the planes that survived. The real method is more useful than the legend.
Abraham Wald, 1943; explained by Marc Mangel and Francisco J. Samaniego, 1984
Chance that a single hit brings the plane down, in Wald's worked example
What he was looking at
During the Second World War, Abraham Wald, a mathematician who had left Austria for the United States, worked with Columbia’s Statistical Research Group on problems for the military. One of them was aircraft vulnerability: where on a bomber does a hit do the most damage?
The only damage anyone could inspect was on the planes that made it home. The planes that were shot down took their bullet holes with them.
What he found
“Undoubtedly, the probability of downing a plane by a hit will depend considerably upon the part that receives the hit.” Wald, Statistical Research Group memo, reprinted in CNA CRC 432 (1980), p. 56
Wald’s insight was that the missing planes still leave a trace. You know how many planes went out and how many came back, and how many hits the survivors took in each area. If hits in one area are rare among the survivors, it’s not because that area rarely gets hit. It’s because the planes hit there mostly didn’t come back.
He turned that into a method for estimating each area’s vulnerability from survivor data alone. In his worked example, using hypothetical numbers, one hit in the engines downed the plane 39% of the time, and one hit in the fuselage 5%.
“for the observed data of this hypothetical example, the engine area is the most vulnerable” p. 65
The memos weren’t published at the time. The Center for Naval Analyses reprinted them in 1980, and in 1984 Marc Mangel and Francisco Samaniego explained the method for statisticians: “Abraham Wald worked on the problem of estimating the vulnerability of aircraft, using data obtained from survivors.”
You may know the popular version, in which Wald tells the Air Force to armor the places with no bullet holes. It’s an embellishment. As the mathematician Bill Casselman has pointed out, the documented record is Wald’s memos and a couple of brief mentions in a colleague’s memoir. The real work is better anyway, because it’s a method, not a hunch.
What it means
A sample of survivors is a filtered sample. What it doesn’t show you is the information. The question isn’t “what do the survivors have in common?” It’s “what would I see if the missing ones were here?”
Where it shows up in a revenue team
- Your current customers are the planes that came back. The ones who churned took their reasons with them.
- “Profile our best customers” built only from today’s book describes who survived, not who you won.
- Win-loss reviews of won deals, surveys of customers who stayed, approvals with no record of the declines: all look only at survivors.
- Count what’s missing. Keep the leads that never converted, the applications you declined and the accounts that left, and analyze them with the rest.