Natural Analysts in Adaptive Data Analysis

Understand

Adaptive data analysis is frequently criticized for its pessimistic generalization guarantees.

• The source of these pessimistic bounds is a model that permits arbitrary, possibly adversarial analysts that optimally use information to bias results.
• While being a central issue in the field, still lacking are notions of natural analysts that allow for more optimistic bounds faithful to the reality that typical analysts aren't adversarial.
• In this work, we propose notions of natural analysts that smoothly interpolate between the optimal non-adaptive bounds and the best-known adaptive generalization bounds.