Fetching the paper…

Reading the bibliography…

2017

Democratizing online controlled experiments at Booking.com

Kaufman, Raphael Lopez, Pitchforth, Jegar, Vermeer, Lukas

Original
PDF
Post on X

Understand

There is an extensive literature about online controlled experiments, both on the statistical methods available to analyze experiment results as well as on the infrastructure built by several large scale Internet companies but also on the organizational challenges of embracing online experiments to inform product development.

At Booking.com we have been conducting evidenced based product development using online experiments for more than ten years.
Our methods and infrastructure were designed from their inception to reflect Booking.com culture, that is, with democratization and decentralization of experimentation and decision making in mind.
In this paper we explain how building a central repository of successes and failures to allow for knowledge sharing, having a generic and extensible code library which enforces a loose coupling between experimentation and business logic, monitoring closely and transparently the quality and the reliability of the data gathering pipelines to build trust in the experimentation infrastructure, and putting in place safeguards to enable anyone to have end to end ownership of their experiments have allowed such a large organization as Booking.com to truly and successfully democratize experimentation.

Democratizing online controlled experiments at Booking.com · Around

Built on

Rubin, Donald B. Estimating causal effects of treatments in randomized and nonrandomized studies
1974
Earlier work this paper cites.
Box, George EP, J. Stuart Hunter, and William Gordon Hunter. Statistics for experimenters: design, innovation, and discovery
2005
Earlier work this paper cites.
Tang, Diane, Ashish Agarwal, Deirdre O’Brien, Mike Meyer. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation
2010
Earlier work this paper cites.
Kohavi, Ron, et al. Online controlled experiments at large scale
2013
Earlier work this paper cites.

Similar

Bakshy, Eytan, Dean Eckles, Michael S. Bernstein. Designing and Deploying Online Field Experiments
2014
Cited alongside, same era.
Xu, Ya, et al. From infrastructure to culture: A/b testing challenges in large scale social networks
2015
Cited alongside, same era.
Xie, Huizhi, and Juliette Aurisset. Improving the sensitivity of online controlled experiments: Case studies at netflix
2016
Cited alongside, same era.
Zhou, Haotian, and Ayelet Fishbach. The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions
2016
Cited alongside, same era.

Then

Fabijan, Aleksander, et al. The Evolution of Continuous Experimentation in Software Product Development
2017
Closest in time.
Silberzahn, Raphael, et al. Many analysts, one dataset: Making transparent how variations in analytical choices affect results
2017
Closest in time.
Munafò, Marcus R., et al. A manifesto for reproducible science
2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…