Research library Measurement, experiments and ad platforms
Your Ad Test Compared Audiences, Not Ads.
Ad platforms show each version in a test to a different, optimized mix of people. Braun and Schwartz showed that can change the size of the result, and even which ad wins.
Michael Braun and Eric M. Schwartz, 2025
Conversion rate for two ads, as the test reports it and with the same audience
What they were looking at
An A/B test on an ad platform looks like a clean experiment: two versions of an ad, the same budget, the same targeting settings, and a winner at the end.
Michael Braun and Eric Schwartz, two marketing professors, pointed at the step most advertisers never see. The platform’s algorithm decides who sees each version, and it learns as it goes. If Ad A starts getting clicks from one kind of person and Ad B from another, the algorithm finds more of each. They call the result divergent delivery. They extended the standard model of cause and effect to account for it, ran simulations, and showed it happening in a real A/B test on a major ad platform.
What they found
“experimenters may not be learning what they think they are learning from ad A/B tests.” Braun & Schwartz (2025), abstract
The two ads end up in front of different audiences, so the test can’t separate what the ad did from whom it was shown to:
“the estimated comparisons confound the effect of the ad content with the effect of algorithmic targeting.” Same
And the damage isn’t limited to a little noise:
“algorithmic targeting, user heterogeneity, and data aggregation conspire to confound the magnitude, and even the sign, of ad A/B test results.” Same
Braun put it simply when the paper came out: “The winning ad may have performed better simply because the algorithm showed it to users who were more prone to respond.”
What it means
On a platform that optimizes delivery, an A/B test is partly a test of the algorithm’s targeting. The winner may be the ad that happened to find better prospects, not the better ad.
The paper’s full title promises both halves: what these tests can’t tell you, and what they can. Read it before you redesign how you test.
Where it shows up in a revenue team
- Creative tests on Meta and Google. The “winning” creative may simply have been shown to better prospects.
- The algorithm chases the conversion you define. If that’s any lead, each ad drifts toward whoever converts cheapest for it, and those are rarely the same people.
- Judge creative on what comes after. Compare the value of the customers each version brought in, a few months on, not just the conversion rate in the test.