All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously

Understand

Variable importance (VI) tools describe how much covariates contribute to a prediction model's accuracy.

• However, important variables for one well-performing model (for example, a linear model $f(\mathbf{x})=\mathbf{x}^{T}\beta$ with a fixed coefficient vector $\beta$) may be unimportant for another model.
• In this paper, we propose model class reliance (MCR) as the range of VI values across all well-performing model in a prespecified class.
• Thus, MCR gives a more comprehensive description of importance by accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well.