A Decision Theoretic Approach to A/B Testing

Understand

A/B testing is ubiquitous within the machine learning and data science operations of internet companies.

• Generically, the idea is to perform a statistical test of the hypothesis that a new feature is better than the existing platform---for example, it results in higher revenue.
• If the p value for the test is below some pre-defined threshold---often, 0.05---the new feature is implemented.
• The difficulty of choosing an appropriate threshold has been noted before, particularly because dependent tests are often done sequentially, leading some to propose control of the false discovery rate (FDR) rather than use of a single, universal threshold.