Rethinking Distribution Shifts: Empirical Analysis and Modeling for Tabular Data

Understand

Different distribution shifts require different interventions, and algorithms must be grounded in the specific shifts they address.

• However, methodological development for robust algorithms typically relies on structural assumptions that lack empirical validation.
• Advocating for an empirically grounded data-driven approach to algorithm development, we build an empirical testbed comprising natural shifts across 8 tabular datasets, 172 distribution pairs over 45 methods and 90,000 method configurations encompassing empirical risk minimization and distributionally robust optimization (DRO) methods.
• We find $Y|X$-shifts are most prevalent in our testbed, in stark contrast to the heavy focus on $X$ (covariate)-shifts in the ML literature, and that the performance of robust algorithms is no better than that of vanilla methods.