2020

iCapsNets: Towards Interpretable Capsule Networks for Text Classification

Wang, Zhengyang, Hu, Xia, Ji, Shuiwang

Understand

Many text classification applications require models with satisfying performance as well as good interpretability.

Traditional machine learning methods are easy to interpret but have low accuracies.
The development of deep learning models boosts the performance significantly.
However, deep learning models are typically hard to interpret.

Built on

S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
T. Joachims, “Text categorization with support vector machines: Learning with many relevant features,” in European Conference on Machine Learning . Springer, 1998, pp. 137–142
1998
Earlier work this paper cites.
Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A neural probabilistic language model,” Journal of Machine Learning Research , vol. 3, no. Feb, pp. 1137–1155, 2003
2003
Earlier work this paper cites.
L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of Machine Learning Research , vol. 9, no. Nov, pp. 2579–2605, 2008
2008
Earlier work this paper cites.
T. Mikolov, M. Karafiát, L. Burget, J. Černockỳ, and S. Khudanpur, “Recurrent neural network based language model,” in Eleventh Annual Conference of the International Speech Communication Association , 2010
2010
Earlier work this paper cites.
R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “Natural language processing (almost) from scratch,” Journal of Machine Learning Research , vol. 12, no. Aug, pp. 2493–2537, 2011
2011
Earlier work this paper cites.
G. E. Hinton, A. Krizhevsky, and S. D. Wang, “Transforming auto-encoders,” in International Conference on Artificial Neural Networks . Springer, 2011, pp. 44–51
2011
Earlier work this paper cites.
T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781 , 2013
Original
2013
Earlier work this paper cites.
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in Neural Information Processing Systems , 2013, pp. 3111–3119
2013
Earlier work this paper cites.
T. Mikolov, W.-t. Yih, and G. Zweig, “Linguistic regularities in continuous space word representations,” in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2013, pp. 746–751
2013
Earlier work this paper cites.
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in Neural Information Processing Systems , 2014, pp. 3104–3112
2014
Earlier work this paper cites.
J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” in NIPS 2014 Workshop on Deep Learning, December 2014 , 2014
2014
Earlier work this paper cites.
J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , 2014, pp. 1532–1543
2014
Earlier work this paper cites.
C. dos Santos and M. Gatti, “Deep convolutional neural networks for sentiment analysis of short texts,” in Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers , 2014, pp. 69–78
2014
Earlier work this paper cites.
N. Kalchbrenner, E. Grefenstette, and P. Blunsom, “A convolutional neural network for modelling sentences,” in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, 2014
2014
Earlier work this paper cites.

Similar

Y. Kim, “Convolutional neural networks for sentence classification,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , 2014, pp. 1746–1751
2014
Cited alongside, same era.
X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” in Advances in Neural Information Processing Systems , 2015, pp. 649–657
2015
Cited alongside, same era.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in International Conference on Learning Representations , 2015
2015
Cited alongside, same era.
R. Johnson and T. Zhang, “Effective use of word order for text categorization with convolutional neural networks,” in Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2015, pp. 103–112
2015
Cited alongside, same era.
D. Tang, B. Qin, and T. Liu, “Document modeling with gated recurrent neural network for sentiment classification,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , 2015, pp. 1422–1432
2015
Cited alongside, same era.
R. Johnson and T. Zhang, “Semi-supervised convolutional neural networks for text categorization via region embedding,” in Advances in Neural Information Processing Systems , 2015, pp. 919–927
2015
Cited alongside, same era.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision , vol. 115, no. 3, pp. 211–252, 2015
2015
Cited alongside, same era.
Y. Xiao and K. Cho, “Efficient character-level document classification by combining convolution and recurrent layers,” arXiv preprint arXiv:1602.00367 , 2016
Original
2016
Cited alongside, same era.
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al. , “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144 , 2016
Original
2016
Cited alongside, same era.
Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, “Hierarchical attention networks for document classification,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2016, pp. 1480–1489
2016
Cited alongside, same era.
A. Conneau, H. Schwenk, L. Barrault, and Y. Lecun, “Very deep convolutional networks for text classification,” in European Chapter of the Association for Computational Linguistics , 2017
2017
Cited alongside, same era.
D. Yogatama, C. Dyer, W. Ling, and P. Blunsom, “Generative and discriminative text classification with recurrent neural networks,” in International Conference on Machine Learning . International Machine Learning Society, 2017
2017
Cited alongside, same era.

Then

S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” in Advances in Neural Information Processing Systems , 2017, pp. 3859–3869
2017
Later among the works it cites.
M. Du, N. Liu, Q. Song, and X. Hu, “Towards explanation of dnn-based prediction with guided feature inversion,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2018, pp. 1358–1367
2018
Later among the works it cites.
C. Qiao, B. Huang, G. Niu, D. Li, D. Dong, W. He, D. Yu, and H. Wu, “A new method of region embedding for text classification,” in International Conference on Learning Representations , 2018
2018
Later among the works it cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Later among the works it cites.
Z. Wang and S. Ji, “Learning convolutional text representations for visual question answering,” in Proceedings of the 2018 SIAM International Conference on Data Mining . SIAM, 2018, pp. 594–602
2018
Later among the works it cites.
L. Xiao, H. Zhang, W. Chen, Y. Wang, and Y. Jin, “Mcapsnet: Capsule network for text with multi-task learning,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018, pp. 4565–4574
2018
Later among the works it cites.
M. Yang, W. Zhao, J. Ye, Z. Lei, Z. Zhao, and S. Zhang, “Investigating capsule networks with dynamic routing for text classification,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018, pp. 3110–3119
2018
Later among the works it cites.
M. Du, N. Liu, and X. Hu, “Techniques for interpretable machine learning,” Communications of the ACM , vol. 63, no. 1, pp. 68–77, 2019
2019
Later among the works it cites.
M. Du, N. Liu, F. Yang, and X. Hu, “Learning credible deep neural networks with rationale regularization,” arXiv preprint arXiv:1908.05601 , 2019
Original
2019
Later among the works it cites.
F. Yang, S. K. Pentyala, S. Mohseni, M. Du, H. Yuan, R. Linder, E. D. Ragan, S. Ji, and X. Hu, “Xfake: explainable fake news detector with visualizations,” in The World Wide Web Conference , 2019, pp. 3600–3604
2019
Later among the works it cites.
K. Shu, L. Cui, S. Wang, D. Lee, and H. Liu, “defend: Explainable fake news detection,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2019, pp. 395–405
2019
Later among the works it cites.
L. Kopitar, L. Cilar, P. Kocbek, and G. Stiglic, “Local vs. global interpretability of machine learning models in type 2 diabetes mellitus screening,” in Artificial Intelligence in Medicine: Knowledge Representation and Transparent and Explainable Systems . Springer, 2019, pp. 108–119
2019
Later among the works it cites.
H. Yuan, Y. Chen, X. Hu, and S. Ji, “Interpreting deep models for text analysis via optimization and regularization methods,” in AAAI-19: Thirty-Third AAAI Conference on Artificial Intelligence . Association for the Advancement of Artificial Intelligence (AAAI), 2019
2019
Later among the works it cites.
A. Rai, “Explainable ai: from black box to glass box,” Journal of the Academy of Marketing Science , vol. 48, no. 1, pp. 137–141, 2020
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…