2020

NBDT: Neural-Backed Decision Trees

Wan, Alvin, Dunlap, Lisa, Ho, Daniel et al.

Understand

Machine learning applications such as finance and medicine demand accurate and justifiable predictions, barring most deep learning methods from use.

In response, previous work combines decision trees with deep learning, yielding models that (1) sacrifice interpretability for accuracy or (2) sacrifice accuracy for interpretability.
We forgo this dilemma by jointly improving accuracy and interpretability using Neural-Backed Decision Trees (NBDTs).
NBDTs replace a neural network's final linear layer with a differentiable sequence of decisions and a surrogate loss.

Built on

Initializing neural networks using decision trees
Arunava Banerjee · 1990
Earlier work this paper cites.
Initializing neural networks using decision trees
Arunava Banerjee · 1994
Earlier work this paper cites.
Using sampling and queries to extract rules from trained neural networks
Mark W Craven and Jude W Shavlik · 1994
Earlier work this paper cites.
Extracting tree-structured representations of trained networks
Mark Craven and Jude W Shavlik · 1996
Earlier work this paper cites.
Extracting decision trees from trained neural networks
R. Krishnan, G. Sivakumar, and P. Bhattacharya · 1999
Earlier work this paper cites.
Converting a trained neural network to a decision tree dectext - decision tree extractor
Olcay Boz · 2000
Earlier work this paper cites.
Decision tree extraction from trained neural networks
Darren Dancey, David McLean, and Zuhair Bandar · 2004
Earlier work this paper cites.
ImageNet: A Large-Scale Hierarchical Image Database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky · 2009
Earlier work this paper cites.
Mnist handwritten digit database
Yann LeCun, Corinna Cortes, and CJ Burges · 2010
Earlier work this paper cites.
A survey of hierarchical classification across different application domains
Carlos N Silla and Alex A Freitas · 2011
Earlier work this paper cites.
Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition
Jia Deng, Jonathan Krause, Alexander C Berg, and Li Fei-Fei · 2012
Earlier work this paper cites.
Deep inside convolutional networks: Visualising image classification models and saliency maps
Original
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman · 2013
Earlier work this paper cites.
Neural decision forests for semantic image labelling
Samuel Rota Bulo and Peter Kontschieder · 2014
Earlier work this paper cites.
Striving for simplicity: The all convolutional net
Original
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller · 2014
Earlier work this paper cites.
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus · 2014
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
Deep neural decision forests
Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo · 2015
Earlier work this paper cites.

Similar

Tiny imagenet visual recognition challenge
Ya Le and Xuan Yang · 2015
Cited alongside, same era.
Network of experts for large-scale image categorization
Karim Ahmed, Mohammadharis Baig, and Lorenzo Torresani · 2016
Cited alongside, same era.
The mythos of model interpretability. corr abs/1606.03490 (2016)
Original
Zachary Chase Lipton · 2016
Cited alongside, same era.
Blockout: Dynamic model selection for hierarchical deep networks
Calvin Murdock, Zhen Li, Howard Zhou, and Tom Duerig · 2016
Cited alongside, same era.
Deep decision network for multi-class image classification
Venkatesh N. Murthy, Vivek Singh, Terrence Chen, R. Manmatha, and Dorin Comaniciu · 2016
Cited alongside, same era.
”why should I trust you?”: Explaining the predictions of any classifier
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin · 2016
Cited alongside, same era.
Monocular depth estimation using neural regression forest
Anirban Roy and Sinisa Todorovic · 2016
Cited alongside, same era.
Top-down neural attention by excitation backprop
Original
Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff · 2016
Cited alongside, same era.
Deep convolutional decision jungle for image classification
Original
Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim · 2017
Cited alongside, same era.
Towards a rigorous science of interpretable machine learning
Original
Finale Doshi-Velez and Been Kim · 2017
Cited alongside, same era.
UCI machine learning repository, 2017
Dheeru Dua and Casey Graff · 2017
Cited alongside, same era.
Distilling a neural network into a soft decision tree
Original
Nicholas Frosst and Geoffrey E. Hinton · 2017
Cited alongside, same era.

Then

Conditional information gain networks
Ufuk Can Biçici, Cem Keskin, and Lale Akarun · 2018
Later among the works it cites.
Cnn-rnn: a large-scale hierarchical image classification framework
Yanming Guo, Yu Liu, Erwin M Bakker, Yuanhao Guo, and Michael S Lew · 2018
Later among the works it cites.
Deep neural network initialization with decision trees
Kelli Humbird, Luc Peterson, and Ryan McClarren · 2018
Later among the works it cites.
Splinenets: Continuous neural decision graphs
Cem Keskin and Shahram Izadi · 2018
Later among the works it cites.
Effectiveness of hierarchical softmax in large scale classification tasks
Abdul Arfat Mohammed and Venkatesh Umaashankar · 2018
Later among the works it cites.
Rise: Randomized input sampling for explanation of black-box models
Vitali Petsiuk, Abir Das, and Kate Saenko · 2018
Later among the works it cites.
Manipulating and measuring model interpretability
F Poursabzi-Sangdeh, D Goldstein, J Hofman, J Vaughan, and H Wallach · 2018
Later among the works it cites.
Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. manuscript based on c. rudin please stop explaining black box machine learning models for high stakes decisions
C Rudin · 2018
Later among the works it cites.
Hydranets: Specialized dynamic architectures for efficient inference
Ravi Teja Mullapudi, William R. Mark, Noam Shazeer, and Kayvon Fatahalian · 2018
Later among the works it cites.
Convolutional networks with adaptive inference graphs
Andreas Veit and Serge Belongie · 2018
Later among the works it cites.
XOC: explainable observer-classifier for explainable binary decisions
Original
Stephan Alaniz and Zeynep Akata · 2019
Later among the works it cites.
Integrating domain knowledge: using hierarchies to improve deep classifiers
Clemens-Alexander Brust and Joachim Denzler · 2019
Later among the works it cites.
Machine learning interpretability: A survey on methods and metrics
Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso · 2019
Later among the works it cites.
Transferring tree ensembles to neural networks
Chapman Siu · 2019
Later among the works it cites.
Adaptive neural trees, 2019
Ryutaro Tanno, Kai Arulkumaran, Daniel C. Alexander, Antonio Criminisi, and Aditya Nori · 2019
Later among the works it cites.
Condconv: Conditionally parameterized convolutions for efficient inference
Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam · 2019
Later among the works it cites.
From local explanations to global understanding with explainable ai for trees, nat. mach. intell., 2, 56–67, 2020
SM Lundberg, G Erion, H Chen, A DeGrave, JM Prutkin, B Nair, R Katz, J Himmelfarb, N Bansal, and S-i Lee · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…