A Common Misassumption in Online Experiments with Machine Learning Models

Understand

Online experiments such as Randomised Controlled Trials (RCTs) or A/B-tests are the bread and butter of modern platforms on the web.

• They are conducted continuously to allow platforms to estimate the causal effect of replacing system variant "A" with variant "B", on some metric of interest.
• These variants can differ in many aspects.
• In this paper, we focus on the common use-case where they correspond to machine learning models.