Metrics and incentives

"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

Once cost per lead becomes the target, its link to revenue starts to break.

"When a measure becomes a target, it ceases to be a good measure." She credits Keith Hoskin (1996), whose version is "every measure which becomes a target becomes a bad measure."

The phrasing everyone quotes, correctly attributed.

"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

Two things get corrupted, the number and the behavior it was meant to measure.

Because models are built from people's decision rules, "any change in policy will systematically alter the structure of econometric models."

Funnel conversion rates estimated from history shift once you change bids, pay or routing, because people respond to the change.

Organisms of every kind "seek information concerning what activities are rewarded." Kerr names four causes of misaligned rewards: fascination with "objective" criteria, overemphasis on highly visible behaviors, hypocrisy, and putting morality or equity ahead of efficiency.

Pay per appointment and you get appointments.

Single, multiple and composite measures all distorted performance. Examples include interviewers judged on interview counts, investigators on case quotas, and production records that led to neglected maintenance.

The oldest management evidence that a single-number target pulls effort toward what's easy to count.

Measured-performance regimes invite "gaming the stats." Metrics help "as a complement to judgment based on personal experience," not as a replacement for it.

The book-length case for metrics plus judgment, rather than metrics instead of judgment.

Four failure types:

• Regressional: "selection for an imperfect proxy necessarily also selects for noise."
• Extremal: old relationships break at the extremes.
• Causal: the act of intervening breaks the link.
• Adversarial: agents game the measure.

Vocabulary for diagnosing which failure a cost-per-lead target triggered.

"The first step is to measure whatever can be easily measured. … The second step is to disregard that which can't be easily measured … The third step is to presume that what can't be measured easily really isn't very important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide."

First delivered to a room of sales executives.

The informativeness principle: a measure is worth paying on if and only if it carries information about the effort you want.

Pay on signals that reveal effort toward value, not on whatever is cheapest to count.

Incentives "direct the allocation of the agents' attention among their various duties." The harder the other tasks are to measure, the weaker the incentive on the measured one should be; sometimes the best contract is a fixed wage.

Strong pay on lead volume pulls effort away from lead quality, which nobody measures.

Pay based on a measure that isn't the firm's real objective falls short even with risk-neutral agents. Every measure has two properties: distortion (how far it points from value) and risk (how noisy it is).

Cost per lead is low-risk and highly distorted. Value measures are noisier but point the right way.

Nonlinear pay creates an "incentive to manipulate prices, influence the timing of customer purchases, and vary effort." Manufacturers' sales are "higher at the end of the fiscal year, and lower at the beginning."

Quota cliffs move when revenue lands, not just how much.

Salespeople "are adept at gaming the timing of deal closure" and accept lower prices in quarters where closing pays more, "resulting in mispricing that costs the vendor 6%–8% of revenue."

A hard number on what accelerators cost.

At a contact-lens maker (87 reps, three years), a redesigned quota, bonus and ratchet plan was implemented and raised revenue 9%, "about $0.98 million incremental revenues per month."

Field evidence that the structure of the plan moves revenue.

• "bonuses enhance productivity across all segments"
• Quarterly bonuses help weak performers
• "overachievement commissions help sustain the high productivity of the best performers"

The counterpoint: well-designed quotas and bonuses work, and different parts of a plan move different reps.

Switching a drug company's sales force from a quota bonus to a commission "increases sales force productivity by approximately 24%," but "induced greater neglect of nonincentivized tasks": daily doctor visits fell from 10.08 to 7.93.

The multitask problem caught in real field data.

"targets and terror" management assumes a measured part can stand for the whole and that gaming doesn't matter. NHS evidence shows both assumptions fail.

A documented case of an organization damaged by its own targets.

• Consent order: the bank "set sales goals and implemented sales incentives." Its own analysis found 1,534,280 deposit accounts "that may not have been authorized" (May 2011–July 2015). About 5,300 employees were fired; the CFPB penalty was $100M.
• Board report: "The root cause of sales practice failures was the distortion of the Community Bank's sales culture and performance management system."

The public, documented extreme of a volume target.