2019

Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance

Hooker, Giles, Mentch, Lucas, Zhou, Siyu

Understand

This paper reviews and advocates against the use of permute-and-predict (PaP) methods for interpreting black box functions.

Methods such as the variable importance measures proposed for random forests, partial dependence plots, and individual conditional expectation plots remain popular because they are both model-agnostic and depend only on the pre-trained model output, making them computationally efficient and widely available in software.
However, numerous studies have found that these tools can produce diagnostics that are highly misleading, particularly when there is strong dependence among features.
The purpose of our work here is to (i) review this growing body of literature, (ii) provide further demonstrations of these drawbacks along with a detailed explanation as to why they occur, and (iii) advocate for alternative measures that involve additional modeling.

Built on

Unbiased measurement of feature importance in tree-based methods
Original
Zhou, Z. and G. Hooker (2019) · 1903
Earlier work this paper cites.
Scalable and efficient hypothesis testing with random forests
Original
Coleman, T., W. Peng, and L. Mentch (2019) · 1904
Earlier work this paper cites.
A debiased mdi feature importance measure for random forests
Original
Li, X., Y. Wang, S. Basu, K. Kumbier, and B. Yu (2019) · 1906
Earlier work this paper cites.
Sensitivity analysis for non-linear mathematical models
Sobol, I. M. (1993) · 1993
Earlier work this paper cites.
Classification with correlated features: unreliability of feature ranking and solutions
Toloşi, L. and T. Lengauer (2011) · 1994
Earlier work this paper cites.
Visualization and exploration of high-dimensional functions using the functional ANOVA decomposition
Roosen, C. B. (1995) · 1995
Earlier work this paper cites.
Random Forests
Breiman, L. (2001) · 2001
Earlier work this paper cites.
Greedy function approximation: a gradient boosting machine
Friedman, J. H. (2001) · 2001
Earlier work this paper cites.
Getting better from worse: Augmented bagging and a cautionary tale of variable importance
Original
Mentch, L. and S. Zhou (2020) · 2003
Earlier work this paper cites.
A unified approach for inference on algorithm-agnostic variable importance
Original
Williamson, B. D., P. B. Gilbert, N. R. Simon, and M. Carone (2020) · 2004
Earlier work this paper cites.
Gene selection and classification of microarray data using random forest
Díaz-Uriarte, R. and S. A. De Andres (2006) · 2006
Earlier work this paper cites.
Testing statistical hypotheses
Lehmann, E. L. and J. P. Romano (2006) · 2006
Earlier work this paper cites.
Generalized additive models: an introduction with R
Wood, S. N. (2006) · 2006
Earlier work this paper cites.
Generalized functional anova diagnostics for high-dimensional functions of dependent variables
Hooker, G. (2007) · 2007
Earlier work this paper cites.
An introduction to copulas
Nelsen, R. B. (2007) · 2007
Earlier work this paper cites.
Bias in random forest variable importance measures: Illustrations, sources and a solution
Strobl, C., A.-L. Boulesteix, A. Zeileis, and T. Hothorn (2007) · 2007
Earlier work this paper cites.

Similar

Controlling variable selection by the addition of pseudovariables
Wu, Y., D. D. Boos, and L. A. Stefanski (2007) · 2007
Cited alongside, same era.
Empirical characterization of random forest variable importance measures
Archer, K. J. and R. V. Kimes (2008) · 2008
Cited alongside, same era.
Conditional variable importance for random forests
Strobl, C., A.-L. Boulesteix, T. Kneib, T. Augustin, and A. Zeileis (2008) · 2008
Cited alongside, same era.
Feature selection with ensembles, artificial variables, and redundancy elimination
Tuv, E., A. Borisov, G. Runger, and K. Torkkola (2009) · 2009
Cited alongside, same era.
The behaviour of random forest permutation-based variable importance measures under predictor correlation
Nicodemus, K. K., J. D. Malley, C. Strobl, and A. Ziegler (2010) · 2010
Cited alongside, same era.
Generalized hoeffding-sobol decomposition for dependent variables-application to sensitivity analysis
Chastaing, G., F. Gamboa, C. Prieur, et al. (2012) · 2012
Cited alongside, same era.
Event labeling combining ensemble detectors and background knowledge
Fanaee-T, H. and J. Gama (2013) · 2013
Cited alongside, same era.
Accurate intelligible models with pairwise interactions
Lou, Y., R. Caruana, J. Gehrke, and G. Hooker (2013) · 2013
Cited alongside, same era.
Deep inside convolutional networks: Visualising image classification models and saliency maps
Original
Simonyan, K., A. Vedaldi, and A. Zisserman (2013) · 2013
Cited alongside, same era.
Sobol’indices and shapley value
Owen, A. B. (2014) · 2014
Cited alongside, same era.
Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation
Goldstein, A., A. Kapelner, J. Bleich, and E. Pitkin (2015) · 2015
Cited alongside, same era.
Grouped variable importance with random forests and application to multiple functional data analysis
Gregorutti, B., B. Michel, and P. Saint-Pierre (2015) · 2015
Cited alongside, same era.

Then

Formal hypothesis tests for additive structure in random forests
Mentch, L. and G. Hooker (2017) · 2017
Later among the works it cites.
Counterfactual explanations without opening the black box: Automated decisions and the gdpr.(2017)
Wachter, S., B. Mittelstadt, and C. Russell (2017) · 2017
Later among the works it cites.
Panning for gold:”model-x” knockoffs for high dimensional controlled variable selection
Candes, E., Y. Fan, L. Janson, and J. Lv (2018) · 2018
Later among the works it cites.
Distribution-free predictive inference for regression
Lei, J., M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman (2018) · 2018
Later among the works it cites.
Auto-encoding knockoff generator for fdr controlled variable selection
Original
Liu, Y. and C. Zheng (2018) · 2018
Later among the works it cites.
Learning global additive explanations for neural nets using model distillation
Original
Tan, S., R. Caruana, G. Hooker, P. Koch, and A. Gordo (2018) · 2018
Later among the works it cites.
Distill-and-compare: Auditing black-box models using transparent model distillation
Tan, S., R. Caruana, G. Hooker, and Y. Lou (2018) · 2018
Later among the works it cites.
All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously
Fisher, A., C. Rudin, and F. Dominici (2019) · 2019
Closest in time.
Standard errors and confidence intervals for variable importance in random forest regression, classification, and survival
Ishwaran, H. and M. Lu (2019) · 2019
Closest in time.
Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead
Rudin, C. (2019) · 2019
Closest in time.
The conditional permutation test for independence while controlling for confounders
Berrett, T. B., Y. Wang, R. F. Barber, and R. J. Samworth (2020) · 2020
Closest in time.
Unbiased variable importance for random forests
Loecher, M. (2020) · 2020
Closest in time.
Fooling lime and shap: Adversarial attacks on post hoc explanation methods
Slack, D., S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju (2020) · 2020
Closest in time.
Mda for random forests: inconsistency, and a practical solution via the sobol-mda
Original
Bénard, C., S. Da Veiga, and E. Scornet (2021) · 2021
Closest in time.
Controlling the false discovery rate via knockoffs
Barber, R. F., E. J. Candès, et al. (2015) · 2085
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…