Overall Evaluation Criterion

What it is

Every AB test ends with a comparison of one number between two groups. The OEC is the name Ron Kohavi and his colleagues at Microsoft gave to that number, or to the weighted combination of numbers that stands in for it. Their 2007 paper says a single metric is highly desired, possibly a weighted combination of objectives, because it forces the trade-offs to be made once, before the test, and aligns the organisation behind a shared definition of success.

The choice is less obvious than it sounds. For a test on a checkout page, the 2007 paper lists conversion rate, units purchased, revenue, profit, expected lifetime value, or a weighted mix of these as candidates. They point in different directions. A discount raises units and conversion and can lower profit. The authors argue that a good OEC "should not be short-term focused (e.g., clicks)" and should include predictors of long-term goals, such as lifetime value and repeat visits. They reject the view that metrics inherently cause short-term thinking, and note that delayed conversions can be captured as well.

The 2012 KDD paper by Kohavi, Deng, Frasca, Longbotham, Walker and Xu returns to the idea with the sharpest example in the literature. A bug in a Bing experiment showed users very poor results, and two organisational metrics rose: distinct queries per user by more than 10 per cent, and revenue per user by more than 30 per cent. The authors explain that degraded organic results push users to issue more queries and click more ads. Distinct queries per user, they say, should be minimised when tasks are completed successfully and not maximised. Revenue per user should not serve as the OEC without constraints on engagement, such as sessions per user. They recommend lifetime customer value as the guiding principle.

In practice an OEC often comes with guardrails: metrics that must not fall while the main metric rises. Kohavi and Thomke (2017) describe a change to Bing's ad headlines that raised revenue by 12 per cent, and report that it did so without hurting key user-experience metrics. The statement shows the structure: revenue was the gain, and the user-experience measures were the check.

In effect

The query count that rose when search got worse

The Bing bug is the standard case. A measure that looked like engagement, distinct queries per user, rose because users could not find what they wanted. A reader looking only at the number would have shipped the bug. The authors' fix is to choose a criterion tied to long-term value and to constrain revenue with engagement measures.

A star widget that cut the ratings

The 2007 paper reports a Microsoft Office Help test in which a yes-or-no rating widget was replaced by a five-star one. Ratings then dropped by an order of magnitude, and the paper records that the team's goal was missed. The example concerns a product team, not a revenue target, and it shows that the measure has to match what the team wants from the change.

Ads that users learn to ignore

Hohnhold, O'Brien and Tang (2015) developed a method for measuring long-term user learning and applied it to ads on Google Search. They found that clicks on ads shift with the quality of the ads users had seen earlier, which the authors call ads blindness and ads sightedness. They built a model that uses short-term metrics to predict long-term outcomes, and applied it to a change in the search ads auction and to a 50 per cent reduction in ad load on Google's mobile search interface. The study is a case of building a better criterion where a single short test misses the effect of adaptation.

A headline worth 12 per cent

In the Bing headline test, revenue rose by 12 per cent without harming key user-experience metrics, in the account Kohavi and Thomke published in 2017. The gain was large enough to trigger a too-good-to-be-true alert, and an analysis confirmed it was real. The account reports user-experience metrics alongside revenue, so the gain was read against a check and not on its own.

What it does not say

An OEC does not remove the need for judgement. A weighted combination of metrics contains a decision about the weights, and the weights are chosen by people. The 2007 paper's argument is that the decision should be made once and in the open, not that it is free of choices.

A proxy for long-term value is a prediction and not a measurement. Hohnhold and colleagues built a model that predicts long-term outcomes from short-term metrics because the long term is not observed during a test. The model's quality sets the limit on the criterion.

An OEC does not by itself show why a variant won. It names the number, and the test reports the difference in it. The explanation of the Bing result came from the authors' analysis of user behaviour after the fact.

The term is also not a claim about psychology or human behaviour, and no replication record applies to it. It is a practice from experimentation teams. This page has not traced uses of the term before the 2007 paper, and the vault records its originator as Kohavi and colleagues with medium confidence.

Who developed it

The term appears in the 2007 KDD paper by Ron Kohavi, Randal Henne and Dan Sommerfield, written at Microsoft, and is developed in the 2012 paper he wrote with five colleagues. The vault records Kohavi and colleagues as originators with medium confidence, because earlier use of the term was not checked.

---