Research library Measurement, experiments and ad platforms
eBay Turned Off Its Search Ads.
In 2012 eBay switched off its brand-name search ads, then its other Google search ads across a third of the US. The experiment found the ads returned a fraction of what the usual numbers said.
Thomas Blake, Chris Nosko and Steven Tadelis, 2015
What they were looking at
eBay spent heavily on search ads. Like most advertisers, it judged them the usual way: track who clicked an ad, then see who bought. Three economists working with eBay’s own data, Thomas Blake, Chris Nosko and Steven Tadelis, asked what would happen if the ads simply weren’t there.
They ran two experiments:
- Brand keywords. In March 2012 eBay stopped bidding on searches for “eBay” on Yahoo! and MSN, and kept bidding on Google for comparison.
- Everything else. It then switched off its non-brand Google search ads for 60 days in about 30% of US markets (68 test areas), and compared them with the rest of the country.
What they found
The brand ads first:
“brand-keyword ads have no measurable short-term benefits.” Blake, Nosko & Tadelis, NBER working paper, abstract
People searching for “eBay” were going to eBay anyway. Without the ad, they clicked the ordinary search result instead.
The non-brand ads did move some people, mostly new and infrequent buyers. But most of the money went somewhere else:
“frequent users whose purchasing behavior is not influenced by ads account for most of the advertising expenses” Same
“returns from paid search are a fraction of conventional non-experimental estimates.” Same
In the working paper’s figures, the experiment put the return on the non-brand ads at about −63%. The usual analysis, which credits the ad for everyone who clicked and bought, put it above 4,100%.
What it means
Attribution answers the question “who clicked an ad before they bought?” The experiment asks a different one: “who bought because of the ad?” For loyal customers who were coming anyway, the answer is almost nobody, and they were where most of the budget went.
It’s also a picture of what a volume-based algorithm will do if you let it. The people most likely to buy are the cheapest conversions to find. Pay for conversions and you’ll keep buying the customers you already had.
Where it shows up in a revenue team
- Brand search, retargeting and last-click reports all credit ads for customers who were on their way already.
- “Our best-performing campaign” is often the one closest to people who’d have bought regardless.
- Run a holdout. Switch the spend off in some markets or for a random slice of the audience, and compare. It’s the only way to see what the ads add.