2020

Towards Interpretable-AI Policies Induction using Evolutionary Nonlinear Decision Trees for Discrete Action Systems

Dhebar, Yashesh, Deb, Kalyanmoy, Nageshrao, Subramanya et al.

Understand

Black-box AI induction methods such as deep reinforcement learning (DRL) are increasingly being used to find optimal policies for a given control task.

Although policies represented using a black-box AI are capable of efficiently executing the underlying control task and achieving optimal closed-loop performance, the developed control rules are often complex and neither interpretable nor explainable.
In this paper, we use a recently proposed nonlinear decision-tree (NLDT) approach to find a hierarchical set of control rules in an attempt to maximize the open-loop performance for approximating and explaining the pre-trained black-box DRL (oracle) agent using the labelled state-action dataset.
Recent advances in nonlinear optimization approaches using evolutionary computation facilitates finding a hierarchical set of nonlinear control rules as a function of state variables using a computationally fast bilevel optimization procedure at each node of the proposed NLDT.

Built on

V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in International Conference on Machine Learning (ICML) , 2016, pp. 1928–1937
1937
Earlier work this paper cites.
J. Kennedy and R. Eberhart, “Particle swarm optimization,” in Proceedings of ICNN’95-International Conference on Neural Networks , vol. 4. IEEE, 1995, pp. 1942–1948
1948
Earlier work this paper cites.
J. L. Bentley, “Multidimensional binary search trees used for associative searching,” Communications of the ACM , vol. 18, no. 9, pp. 509–517, 1975
1975
Earlier work this paper cites.
G. A. Rummery and M. Niranjan, “On-line q-learning using connectionist systems,” University of Cambridge, Department of Engineering Cambridge, UK, Tech. Rep. CUED/F-INFENG/TR166, 1994
1994
Earlier work this paper cites.
K. Deb and R. B. Agrawal, “Simulated binary crossover for continuous search space,” Complex systems , vol. 9, no. 2, pp. 115–148, 1995
1995
Earlier work this paper cites.
R. S. Sutton, “Generalization in reinforcement learning: Successful examples using sparse coarse coding,” in Advances in Neural Information Processing Systems 8 , D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Eds. MIT Press, 1996, pp. 1038–1044
1996
Earlier work this paper cites.
W. D. Smart and L. P. Kaelbling, “Practical reinforcement learning in continuous spaces,” in International Conference on Machine Learning (ICML) , 2000, pp. 903–910
2000
Earlier work this paper cites.
L. Bobrowski and M. Kretowski, “Induction of multivariate decision trees by using dipolar criteria,” in European Conference on Principles of Data Mining and Knowledge Discovery . Springer, 2000, pp. 331–336
2000
Earlier work this paper cites.
M. Kretowski, “An evolutionary algorithm for oblique decision tree induction,” in International Conference on Artificial Intelligence and Soft Computing . Springer, 2004, pp. 432–437
2004
Earlier work this paper cites.
D. Ernst, P. Geurts, and L. Wehenkel, “Tree-based batch mode reinforcement learning,” Journal of Machine Learning Research , vol. 6, no. Apr, pp. 503–556, 2005
2005
Earlier work this paper cites.
M. Kretowski and M. Grześ, “Global induction of oblique decision trees: an evolutionary approach,” in Intelligent Information Processing and Web Mining . Springer, 2005, pp. 309–318
2005
Earlier work this paper cites.
K. Deb, Multi-objective optimization using evolutionary algorithms . Wiley, 2005
2005
Earlier work this paper cites.
K. Deb and A. Srinivasan, “Innovization: Innovating design principles through optimization,” in Proceedings of the 8th annual conference on Genetic and evolutionary computation . ACM, 2006, pp. 1629–1636
2006
Earlier work this paper cites.
J. Peters, K. Mülling, and Y. Altun, “Relative entropy policy search,” in Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-10) , vol. 10. Atlanta, 2010, pp. 1607–1612
2010
Earlier work this paper cites.

Similar

S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , 2011, pp. 627–635
2011
Cited alongside, same era.
F. Maes, R. Fonteneau, L. Wehenkel, and D. Ernst, “Policy search in a space of simple closed-form formulas: towards interpretability of reinforcement learning,” in International Conference on Discovery Science . Springer, 2012, pp. 37–51
2012
Cited alongside, same era.
K. Senda, S. Hattori, T. Hishinuma, and T. Kohda, “Acceleration of reinforcement learning by policy evaluation using nonstationary iterative method,” IEEE transactions on cybernetics , vol. 44, no. 12, pp. 2696–2705, 2014
2014
Cited alongside, same era.
M. L. Koga, V. Freire, and A. H. Costa, “Stochastic abstract policies: Generalizing knowledge to improve reinforcement learning,” IEEE Transactions on Cybernetics , vol. 45, no. 1, pp. 77–88, 2014
2014
Cited alongside, same era.
X. Xu, Z. Huang, D. Graves, and W. Pedrycz, “A clustering-based graph laplacian framework for value function approximation in reinforcement learning,” IEEE Transactions on Cybernetics , vol. 44, no. 12, pp. 2613–2625, 2014
2014
Cited alongside, same era.
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in International Conference on Machine Learning (ICML) , 2015, pp. 1889–1897
2015
Cited alongside, same era.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015
Original
2015
Cited alongside, same era.
H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning” arXiv preprint arXiv:1509.06461 , 2015
Original
2015
Cited alongside, same era.
G. Vandewiele, O. Janssens, F. Ongenae, F. De Turck, and S. Van Hoecke, “Genesim: Genetic extraction of a single, interpretable model,” arXiv preprint arXiv:1611.05722 , 2016
Original
2016
Cited alongside, same era.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017
Original
2017
Cited alongside, same era.
B. Goodman and S. Flaxman, “European union regulations on algorithmic decision-making and a “right to explanation”,” AI magazine , vol. 38, no. 3, pp. 50–57, 2017
2017
Cited alongside, same era.
D. Hein, A. Hentschel, T. Runkler, and S. Udluft, “Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies,” Engineering Applications of Artificial Intelligence , vol. 65, pp. 87–98, 2017
2017
Cited alongside, same era.

Then

Z. C. Lipton, “The mythos of model interpretability,” Queue , vol. 16, no. 3, pp. 31–57, 2018
2018
Later among the works it cites.
M. A. Ahmad, C. Eckert, and A. Teredesai, “Interpretable machine learning in healthcare,” in Proceedings of the 2018 ACM international conference on bioinformatics, computational biology, and health informatics , 2018, pp. 559–560
2018
Later among the works it cites.
R. Noothigattu, D. Bouneffouf, N. Mattei, R. Chandra, P. Madan, K. Varshney, M. Campbell, M. Singh, and F. Rossi, “Interpretable multi-objective reinforcement learning through policy orchestration,” arXiv preprint arXiv:1809.08343 , 2018
Original
2018
Later among the works it cites.
D. Hein, S. Udluft, and T. A. Runkler, “Interpretable policies for reinforcement learning by genetic programming,” Engineering Applications of Artificial Intelligence , vol. 76, pp. 158–169, 2018
2018
Later among the works it cites.
G. Liu, O. Schulte, W. Zhu, and Q. Li, “Toward interpretable deep reinforcement learning with linear model u-trees,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Springer, 2018, pp. 414–429
2018
Later among the works it cites.
A. Verma, V. Murali, R. Singh, P. Kohli, and S. Chaudhuri, “Programmatically interpretable reinforcement learning,” arXiv preprint arXiv:1804.02477 , 2018
Original
2018
Later among the works it cites.
O. Bastani, Y. Pu, and A. Solar-Lezama, “Verifiable reinforcement learning via policy extraction,” in Advances in neural information processing systems (NIPS) , 2018, pp. 2494–2504
2018
Later among the works it cites.
A. Sinha, P. Malo, and K. Deb, “A review on bilevel optimization: From classical to evolutionary approaches and applications,” IEEE Transactions on Evolutionary Computation , vol. 22, no. 2, pp. 276–295, 2018
2018
Later among the works it cites.
W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu, “Interpretable machine learning: definitions, methods, and applications,” arXiv preprint arXiv:1901.04592 , 2019
Original
2019
Later among the works it cites.
S. Nageshrao, B. Costa, and D. Filev, “Interpretable approximation of a deep reinforcement learning agent as a set of if-then rules,” in 18th IEEE International Conference On Machine Learning And Applications (ICMLA) . IEEE, 2019, pp. 216–221
2019
Later among the works it cites.
A. Silva, M. Gombolay, T. Killian, I. Jimenez, and S.-H. Son, “Optimization methods for interpretable differentiable decision trees applied to reinforcement learning,” in International Conference on Artificial Intelligence and Statistics , 2020, pp. 1855–1865
2020
Closest in time.
Y. Dhebar and K. Deb, “Interpretable rule discovery through bilevel optimization of split-rules of nonlinear decision trees for classification problems,” IEEE Transactions on Cybernetics , pp. 1–12, 2020
2020
Closest in time.
E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org , 2016–2020
2020
Closest in time.
P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Methods , vol. 17, pp. 261–272, 2020
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…