Ron Kohavi

Who they are

Kohavi's home page at Stanford lists a PhD in machine learning from Stanford University, where he led the MLC++ project, and a first degree from the Technion in Israel. Before experimentation he held roles at Silicon Graphics, where he managed the MineSet project, and at Blue Martini Software as vice president of business intelligence. At Amazon he was director of data mining and personalisation, and the 2017 Harvard Business Review article records that he was responsible for Weblab, Amazon's experimentation system.

He joined Microsoft in 2005 and founded its Experimentation Platform team in 2006. His page says he rose to Technical Fellow and Corporate Vice President in the Cloud and AI group and led an Analysis and Experimentation group of about 110 data scientists and developers. He later became Vice President and Technical Fellow at Airbnb, working on search relevance, personalisation and experimentation, until March 2021. The page states that his Airbnb team delivered improvements of more than 6 per cent to booking conversion. These details are self-reported on his own page and were not checked against another source.

The page also lists an Individual Lifetime Achievement Award for Experimentation Culture from September 2020, and records that an early version of a paper on online experimentation at Microsoft took third place in the Third Workshop on Data Mining Case Studies and Practice Prize in 2009.

Key papers

Key ideas

A good criterion looks past the short term. The 2007 paper says an overall evaluation criterion "should not be short-term focused (e.g., clicks)" and should include predictors of long-term goals such as lifetime value and repeat visits. The 2012 paper makes the point with the Bing bug, where degraded results raised distinct queries per user by more than 10 per cent and revenue per user by more than 30 per cent. The authors recommend lifetime customer value as the guiding principle. See Overall Evaluation Criterion.

Extreme results call for suspicion. A slide from his 2012 talk tells the reader to find the flaw when results look extreme. In the Bing headline test the revenue jump triggered a too-good-to-be-true alert before an analysis confirmed it was real.

Early readings deceive. The 2012 paper shows an A/A test in which two identical groups appeared to diverge early and then settled near zero. The authors advise running a test for at least a week to capture day-of-week effects, and longer where novelty or primacy effects are a concern.

Most ideas do not win. His 2012 slides report that about a third of the ideas tested at Microsoft were positive and statistically significant, a third were flat and a third were negative and significant. He adds, on the same slide, that intuition is poor: 60 to 90 per cent of ideas do not improve the metric they were designed to improve, depending on the domain.

Opinion should not outrank data. The HiPPO, the highest-paid person's opinion, is the foil in his talk titles and in the 2007 paper, which cites Amazon's culture, where data outweighs intuition, as a model. See AB Testing.

---