2021

Recurrent Neural Networks with Mixed Hierarchical Structures for Natural Language Processing

Luo, Zhaoxin, Zhu, Michael

Understand

Hierarchical structures exist in both linguistics and Natural Language Processing (NLP) tasks.

How to design RNNs to learn hierarchical representations of natural languages remains a long-standing challenge.
In this paper, we define two different types of boundaries referred to as static and dynamic boundaries, respectively, and then use them to construct a multi-layer hierarchical structure for document classification tasks.
In particular, we focus on a three-layer hierarchical structure with static word- and sentence- layers and a dynamic phrase-layer.

J. Schmidhuber, “Neural sequence chunkers,” 1991
1991
Earlier work this paper cites.
J. Schmidhuber, “Learning complex, extended sequences using the principle of history compression,” Neural Computation , vol. 4, no. 2, pp. 234–242, 1992
1992
Earlier work this paper cites.
S. El Hihi and Y. Bengio, “Hierarchical recurrent neural networks for long-term dependencies,” in Advances in neural information processing systems , 1996, pp. 493–499
1996
Earlier work this paper cites.
T. Lin, B. G. Horne, P. Tino, and C. L. Giles, “Learning long-term dependencies in narx recurrent neural networks,” IEEE Transactions on Neural Networks , vol. 7, no. 6, pp. 1329–1338, 1996
1996
Earlier work this paper cites.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.

M. Sahami, S. Dumais, D. Heckerman, and E. Horvitz, “A bayesian approach to filtering junk e-mail,” in Learning for Text Categorization: Papers from the 1998 workshop , vol. 62. Madison, Wisconsin, 1998, pp. 98–105
1998
Cited alongside, same era.
S. Wang and C. D. Manning, “Baselines and bigrams: Simple, good sentiment and topic classification,” in Proceedings of the 50th annual meeting of the association for computational linguistics: Short papers-volume 2 . Association for Computational Linguistics, 2012, pp. 90–94
2012
Cited alongside, same era.
T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781 , 2013
Original
2013
Cited alongside, same era.
N. Kalchbrenner, E. Grefenstette, and P. Blunsom, “A convolutional neural network for modelling sentences,” arXiv preprint arXiv:1404.2188 , 2014
Original
2014
Cited alongside, same era.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473 , 2014
Original
2014
Cited alongside, same era.

J. Koutnik, K. Greff, F. Gomez, and J. Schmidhuber, “A clockwork rnn,” arXiv preprint arXiv:1402.3511 , 2014
Original
2014
Later among the works it cites.
A. Sordoni, Y. Bengio, H. Vahabi, C. Lioma, J. Grue Simonsen, and J.-Y. Nie, “A hierarchical recurrent encoder-decoder for generative context-aware query suggestion,” in Proceedings of the 24th ACM International on Conference on Information and Knowledge Management . ACM, 2015, pp. 553–562
2015
Later among the works it cites.
W. Ling, I. Trancoso, C. Dyer, and A. W. Black, “Character-based neural machine translation,” arXiv preprint arXiv:1511.04586 , 2015
Original
2015
Later among the works it cites.
Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, “Hierarchical attention networks for document classification,” in Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies , 2016, pp. 1480–1489
2016
Later among the works it cites.
J. Chung, S. Ahn, and Y. Bengio, “Hierarchical multiscale recurrent neural networks,” arXiv preprint arXiv:1609.01704 , 2016
Original
2016
Later among the works it cites.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…