Heckman (1979)
James J. Heckman, "Sample Selection Bias as a Specification Error," Econometrica 47, no. 1 (1979): 153–161. doi:10.2307/1912352
What it found
Bias from "non-randomly selected samples" is a form of omitted-variable bias, and it can be corrected with a two-step estimator. This is part of the work behind his 2000 Nobel ("for his development of theory and methods for analyzing selective samples").
Why it matters
A model trained only on approved or closed deals inherits the selection that produced them.
Lakkaraju, Kleinberg, Leskovec, Ludwig & Mullainathan (2017)
"The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables," Proceedings of KDD '17 (2017): 275–284. doi:10.1145/3097983.3098066
What it found
When outcomes are seen only for cases a human approved (bail, loans), you can still evaluate a model by using the differences between lenient and strict decision-makers ("contraction").
Why it matters
You never see outcomes for the files you declined or the leads reps skipped. Differences between underwriters or reps are your natural experiment.
Hand & Henley (1993; 1997): reject inference
David J. Hand and William E. Henley, "Can Reject Inference Ever Work?" IMA Journal of Mathematics Applied in Business and Industry 5, no. 1 (1993): 45–55, doi:10.1093/imaman/5.1.45; and "Statistical Classification Methods in Consumer Credit Scoring: A Review," Journal of the Royal Statistical Society, Series A 160, no. 3 (1997): 523–541, doi:10.1111/j.1467-985X.1997.00078.x.
What it found
Credit scorecards are built only on approved applicants. "Reject inference" tries to guess how the rejected would have performed, and the 1993 paper asks whether it can ever work. The 1997 paper is the standard review of scoring methods.
Why it matters
The lending industry's name for "you only see the loans you made."
Capen, Clapp & Campbell (1971); Thaler (1988): the winner's curse
E. C. Capen, R. V. Clapp and W. M. Campbell, "Competitive Bidding in High-Risk Situations," Journal of Petroleum Technology 23, no. 6 (1971): 641–653, doi:10.2118/2993-PA; Richard H. Thaler, "Anomalies: The Winner's Curse," Journal of Economic Perspectives 2, no. 1 (1988): 191–202, doi:10.1257/jep.2.1.191.
What it found
"a lease winner tends to be the bidder who most overestimates reserves potential." Thaler presents evidence that "the winner's curse may be a common phenomenon."
Why it matters
Winning a contested lead, keyword or media auction is bad news about your own estimate. A/B-test winners suffer the same curse (see Lee and Shen, 2018).
Simpson (1951); Bickel, Hammel & O'Connell (1975)
E. H. Simpson, "The Interpretation of Interaction in Contingency Tables," Journal of the Royal Statistical Society, Series B 13, no. 2 (1951): 238–241, doi:10.1111/j.2517-6161.1951.tb00088.x; P. J. Bickel, E. A. Hammel and J. W. O'Connell, "Sex Bias in Graduate Admissions: Data from Berkeley," Science 187, no. 4175 (1975): 398–404, doi:10.1126/science.187.4175.398.
What it found
Berkeley's combined admissions showed "a clear but misleading pattern of bias against female applicants." Department by department, the data showed "a small but statistically significant bias in favor of women." Women had applied more to the departments that admitted fewer of anyone.
Why it matters
A blended rate (churn, conversion, approval) can move the opposite way from every segment inside it, because the mix changed.
Note. The name "Simpson's paradox" came later (Blyth, 1972).
Read our explainer
Berkson (1946)
Joseph Berkson, "Limitations of the Application of Fourfold Table Analysis to Hospital Data," Biometrics Bulletin 2, no. 3 (1946): 47–53. doi:10.2307/3002000
What it found
Studying only hospital patients can create "a spurious negative association" between a disease and a risk factor (as summarized secondarily; the paper has no abstract).
Why it matters
Analyze only the deals you won and you'll find trade-offs that don't exist in the market.
Mangel & Samaniego (1984): Wald's bombers
Marc Mangel and Francisco J. Samaniego, "Abraham Wald's Work on Aircraft Survivability," Journal of the American Statistical Association 79, no. 386 (1984): 259–267. doi:10.1080/01621459.1984.10478038
What it found
An exposition of Wald's wartime memos on estimating aircraft vulnerability "using data obtained from survivors."
Why it matters
Your surviving customers hide the reasons the others left.
Note. A caution: the popular version of this story (armor the places with no bullet holes) is embellished. This paper is the documented account of Wald's method.
Read our explainer