Research library Humans, algorithms and who decides
The Algorithm Worked. It Was Aimed at the Wrong Thing.
A widely used health algorithm chose patients for extra care by predicting their medical costs. Changing what it predicted would have more than doubled the share of Black patients it flagged for help.
Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, 2019
Percentage of Black patients receiving additional help
What they were looking at
Hospitals and insurers use algorithms to decide which patients get extra care: a nurse who checks in, closer follow-up, help managing a chronic condition. Ziad Obermeyer and his colleagues audited one widely used commercial algorithm of this kind.
Their data came from primary-care patients at a large academic hospital, 2013 to 2015: 6,079 Black patients and 43,539 White patients. Patients above the 97th percentile of the algorithm’s risk score were enrolled in the program automatically, and those above the 55th were referred to their doctor to consider.
What they found
“At a given risk score, Black patients are considerably sicker than White patients” Obermeyer et al. (2019), abstract
The cause was what the algorithm was told to predict:
“The bias arises because the algorithm predicts health care costs rather than illness” Same
Less money is spent on Black patients with the same level of need, so a prediction of cost ranked them as healthier than they were. Remedying that, the authors estimate, would raise the percentage of Black patients receiving extra help from 17.7% to 46.5%. Their general warning:
“the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias” Same
What it means
Nothing in the algorithm was broken. It predicted cost very well. Cost was a convenient, measurable stand-in for need, and the gap between the two is where the harm came from.
This case is about people’s health and about race, and it deserves to be read on its own terms first. It also happens to be the clearest published example of a problem every team that uses an algorithm faces: the model does exactly what it’s pointed at, so the choice of target is the most important decision in the system.
Where it shows up in a revenue team
- Lead volume, conversion count and cost per lead are convenient proxies for value.
- An ad platform told to find conversions will find conversions, very well. Whether they’re worth anything depends on what you told it a conversion is.
- Fix the target, not just the model. Define the conversion around the outcome you actually want, and check that the people it finds look like your best customers.