Measurement, experiments and ad platforms

At eBay, "brand keyword ads have no measurable short-term benefits." For non-brand keywords, frequent users whose behavior ads don't change "account for most of the advertising expenses, resulting in average returns that are negative."

Attribution credits ads for buyers who were coming anyway.

Across 25 field experiments, "the median confidence interval on return on investment is over 100 percentage points wide." Informative tests "can easily require more than 10 million person-weeks," and selection bias "is a crippling concern" for observational methods.

Be suspicious of any precise ROAS number, especially when revenue is heavy-tailed.

Across 15 experiments, "observational methods often fail to accurately recover the treatment effects" of randomized ad experiments, usually by overstating them.

Models built on platform data don't replace a holdout.

In a retailer's retargeting campaign, "the ads lifted website visits by 17.2% and purchases by 10.5%." Ghost ads measure lift just as precisely "while spending at least an order of magnitude less."

A cheaper way to test retargeting, the channel most distorted by selection.

Platforms show each ad version to "distinct and undetectably optimized mixes of users." That mixes up the effect of the ad with the effect of the targeting, and can distort "the magnitude, and even the sign" of the result.

Your winning creative may just have been shown to better prospects. The platform's targeting can decide an ad test before the ad does.

"both the advertiser's budget and the content of the ad each significantly contribute to the skew of Facebook's ad delivery," even "despite neutral targeting parameters."

The platform, not your targeting settings, decides who sees the ad.

Fewer women saw a gender-neutral ad "because younger women are a prized demographic and are more expensive to show ads to." The skew came from auction prices, not targeting choices.

A cost-minimizing algorithm drifts toward whoever is cheapest to reach, whatever they're worth.

The overall evaluation criterion (OEC) "must be measurable in the short term (the duration of an experiment) yet believed to causally drive long-term strategic objectives."

The metric you test on decides which customers you select.

At Google, "the short-term effect is not always predictive of the long-term effect." Users learned to ignore low-quality ads, and the work led to "a 50% reduction of the ad load on Google's mobile search interface."

Short-term revenue wins can teach customers to tune you out.

Combining several short-term outcomes into one index, the first six quarters of a job-training experiment recovered nine-year effects "with a 35% reduction in standard errors."

The rigorous version of "signals": early indicators, validated against long-term value.

A valid surrogate must "reliably predict the overall effect on the clinical outcome. In practice, this requirement frequently fails."

MQLs, CTR and form fills are surrogates. Validate them against revenue before optimizing to them.

The drugs suppressed irregular heartbeats (the surrogate) but were stopped "because of excess mortality": 43 arrhythmic deaths on the drugs against 16 on placebo.

The starkest case on record of a better number and a worse outcome.

Teams launch only the experiments that won, so summing their measured wins overstates the total impact. The authors estimate the bias and correct for it in Airbnb's experiment reporting.

Your testing program's claimed cumulative lift is inflated.

Firms that tried to contact leads within an hour were "nearly seven times as likely to qualify the lead" as those that waited an hour longer, and "more than 60 times as likely" as those that waited 24 hours or more.

Speed matters. The answer to bad routing isn't to ignore speed; it's to route on value with a short time-out.