Research library Selection bias and paradoxes
The Total Can Point the Wrong Way.
In 1973 Berkeley's graduate admissions looked biased against women. Department by department, they weren't. The difference was the mix, and the same thing happens in your churn rate.
E. H. Simpson, 1951; P. J. Bickel, E. A. Hammel and J. W. O'Connell, 1975
Churn by tier, last year and this year
What they were looking at
In the fall of 1973, 8,442 men and 4,321 women applied to graduate programs at the University of California, Berkeley. About 44% of the men were admitted and about 35% of the women.
Eugene Hammel, an associate dean, looked into the numbers because, as the statistician Peter Bickel later recalled, “he was worried that the government would cut funding, on the grounds that Berkeley was biased.” Bickel, Hammel and J. W. O’Connell went through the data department by department. (The often-told story that Berkeley was sued over it appears to be unsupported.)
What they found
The combined figures:
“shows a clear but misleading pattern of bias against female applicants.” Bickel, Hammel & O’Connell (1975), abstract
Taken department by department and combined properly, the picture reversed: “there is a small but statistically significant bias in favor of women.” Women had applied in larger numbers to the departments that admitted a smaller share of everyone who applied, men included.
“The bias in the aggregated data stems not from any pattern of discrimination on the part of admissions committees” Same
The statistician E. H. Simpson had described the trap in 1951, in a short note on how tables of counts combine. Bickel remembered giving Hammel “an example of what I later learned was the Simpson paradox.” The name itself came later.
What it means
A total is an average of its parts, weighted by how many are in each part. Change the weights and the total can move when none of the parts does, or move the opposite way from every one of them.
That’s the chart above. Both tiers churn less this year. The book churns more, because this year’s growth came from the tier that always churns most.
Where it shows up in a revenue team
- Churn. The blended rate can worsen while every tier improves, or improve while every tier gets worse.
- Conversion rates. A channel’s rate “drops” because it’s now sending more of a segment that always converts lower.
- Approval rates, win rates, rep rankings. A rep with tougher territory can lose to a weaker rep on the total and beat them in every segment.
- The fix is simple: break the rate out by tier or segment before you act on it, and check whether the mix moved.