Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Panel Experiments

Understand

Randomized experiments have become the standard method for companies to evaluate the performance of new products or services.

• Beyond aiding managerial decision-making, experiments mitigate risk by limiting the proportion of customers exposed to innovations.
• Since many experiments are conducted sequentially over time, an emerging strategy to further derisk the process is to allow managers to ``peek'' at the results as new data become available and stop the test if the results are statistically significant.
• The class of statistical methods that allow managers to peek and still provide valid inference are often called anytime-valid since they maintain proper uniform type-1 error guarantees.