FlaPy: Mining Flaky Python Tests at Scale

Understand

Flaky tests obstruct software development, and studying and proposing mitigations against them has therefore become an important focus of software engineering research.

• To conduct sound investigations on test flakiness, it is crucial to have large, diverse, and unbiased datasets of flaky tests.
• A common method to build such datasets is by rerunning the test suites of selected projects multiple times and checking for tests that produce different outcomes.
• While using this technique on a single project is mostly straightforward, applying it to a large and diverse set of projects raises several implementation challenges such as (1) isolating the test executions, (2) supporting multiple build mechanisms, (3) achieving feasible run times on large datasets, and (4) analyzing and presenting the test outcomes.