Research library Measurement, experiments and ad platforms
The Drugs Fixed the Number. More Patients Died.
In the 1980s, heart-attack patients were given drugs that suppressed irregular heartbeats, a warning sign of sudden death. A trial found the drugs improved the number and worsened the outcome.
Debra S. Echt et al., 1991 (the CAST trial); Thomas R. Fleming and David L. DeMets, 1996
Deaths from arrhythmia in the trial
What they were looking at
After a heart attack, many patients have irregular heartbeats, and those patients are more likely to die suddenly. In the 1980s there were drugs that suppressed the irregular beats. The logic looked airtight: suppress the beats and you prevent the deaths.
The Cardiac Arrhythmia Suppression Trial, known as CAST, was set up to check that logic:
“designed to test the hypothesis that suppression of ventricular ectopy after a myocardial infarction reduces the incidence of sudden death” Echt et al. (1991), abstract
Patients whose irregular beats could be suppressed by the drugs were randomly assigned to a drug or a placebo.
What they found
The drugs did what they were meant to do to the heartbeat. The patients taking them died more often.
“The use of encainide and flecainide was discontinued because of excess mortality.” Same
Of the 1,498 patients in those two parts of the trial, 43 on the drugs died of arrhythmia, against 16 on placebo. The preliminary report in 1989 counted all deaths: 56 of 730 patients on the drugs, against 22 of 725 on placebo.
Five years later the statisticians Thomas Fleming and David DeMets gathered examples from several areas of medicine to ask how often it gets misled this way. A surrogate end point, the number you can measure now in place of the outcome you care about, has to reliably predict the effect on that outcome:
“In practice, this requirement frequently fails.” Fleming & DeMets (1996), abstract
What it means
A surrogate is only safe to optimize if moving it moves the outcome. Here the drugs moved the surrogate the right way and the outcome the wrong way, and nobody could see that until a trial counted the outcome itself.
This is the starkest example on record, and it involves real deaths, so it’s worth being careful with. The lesson for the rest of us is narrower but real: an improved proxy proves nothing on its own.
Where it shows up in a revenue team
- MQLs, click-through rate, form fills, cost per lead and approval rate are all surrogates for revenue.
- Validate each against the outcome before you optimize to it. If the leads that pass your score don’t turn into better customers, the score is a suppressed heartbeat.
- Keep counting the outcome. Track what each month’s new customers actually return over time, not just how the leading indicators moved.