2019

Improving Sequence Modeling Ability of Recurrent Neural Networks via Sememes

Qin, Yujia, Qi, Fanchao, Ouyang, Sicong et al.

Understand

Sememes, the minimum semantic units of human languages, have been successfully utilized in various natural language processing applications.

However, most existing studies exploit sememes in specific tasks and few efforts are made to utilize sememes more fundamentally.
In this paper, we propose to incorporate sememes into recurrent neural networks (RNNs) to improve their sequence modeling ability, which is beneficial to all kinds of downstream tasks.
We design three different sememe incorporation methods and employ them in typical RNNs including LSTM, GRU and their bidirectional variants.

Built on

L. Bloomfield, “A set of postulates for the science of language,” Language , vol. 2, no. 3, pp. 153–164, 1926
1926
Earlier work this paper cites.
D. E. Rumelhart, G. E. Hinton, R. J. Williams et al. , “Learning representations by back-propagating errors,” Cognitive modeling , vol. 5, no. 3, p. 1, 1988
1988
Earlier work this paper cites.
J. Schmidhuber, “Learning complex, extended sequences using the principle of history compression,” Neural Computation , vol. 4, no. 2, pp. 234–242, 1992
1992
Earlier work this paper cites.
M. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a large annotated corpus of english: The penn treebank,” Computational Linguistics , vol. 19, no. 2, 1993
1993
Earlier work this paper cites.
A. Wierzbicka, Semantics: Primes and universals: Primes and universals . Oxford University Press, UK, 1996
1996
Earlier work this paper cites.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
G. Miller, WordNet: An electronic lexical database . MIT press, 1998
1998
Earlier work this paper cites.
Q. Liu and S. Li, “Word similarity computing based on HowNet,” International Journal of Computational Linguistics & Chinese Language Processing , vol. 7, no. 2, pp. 59–76, 2002
2002
Earlier work this paper cites.
Z. Dong and Q. Dong, “HowNet-a hybrid language and knowledge resource,” in Proceedings of NLP-KE , 2003
2003
Earlier work this paper cites.
M. Hu and B. Liu, “Mining and summarizing customer reviews,” in Proceedings of KDD , 2004
2004
Earlier work this paper cites.
B. Dolan, C. Quirk, and C. Brockett, “Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources,” in Proceedings of COLING , 2004
2004
Earlier work this paper cites.
X. Duan, J. Zhao, and B. Xu, “Word sense disambiguation through sememe labeling.” in Proceedings of IJCAI , 2007
2007
Earlier work this paper cites.
T. Mikolov, M. Karafiát, L. Burget, J. Černockỳ, and S. Khudanpur, “Recurrent neural network based language model,” in Proceedings of Eleventh Annual Conference of the International Speech Communication Association , 2010
2010
Earlier work this paper cites.
X. Fu, G. Liu, Y. Guo, and Z. Wang, “Multi-aspect sentiment analysis for chinese online social reviews based on topic modeling and HowNet lexicon,” Knowledge-Based Systems , vol. 37, pp. 186–195, 2013
2013
Earlier work this paper cites.
R. Speer and C. Havasi, “ConceptNet 5: A large semantic network for relational knowledge,” in The People’s Web Meets NLP . Springer, 2013, pp. 161–176
2013
Earlier work this paper cites.
J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of EMNLP , 2014
2014
Earlier work this paper cites.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in Proceedings of ICLR , 2014
2014
Earlier work this paper cites.
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” in Proceedings of EMNLP , 2014
2014
Earlier work this paper cites.

Similar

S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” in Proceedings of EMNLP , 2015
2015
Cited alongside, same era.
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in Proceedings of ICLR , 2015
2015
Cited alongside, same era.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in Proceedings of ICLR , 2015
2015
Cited alongside, same era.
P. Bachman and D. Precup, “Variational generative stochastic networks with collaborative shaping.” in Proceedings of ICML , 2015
2015
Cited alongside, same era.
S. Ahn, H. Choi, T. Pärnamaa, and Y. Bengio, “A neural knowledge language model,” arXiv preprint arXiv:1608.00318 , 2016
Original
2016
Cited alongside, same era.
S. R. Bowman, J. Gauthier, A. Rastogi, R. Gupta, C. D. Manning, and C. Potts, “A fast unified model for parsing and sentence understanding,” in Proceedings of ACL , 2016
2016
Cited alongside, same era.
L. Mou, R. Men, G. Li, Y. Xu, L. Zhang, R. Yan, and Z. Jin, “Natural language inference by tree-based convolution and heuristic matching,” in Proceedings of ACL , 2016
2016
Cited alongside, same era.
P. Nakov, A. Ritter, S. Rosenthal, F. Sebastiani, and V. Stoyanov, “Semeval-2016 task 4: Sentiment analysis in twitter,” in Proceedings of SemEval , 2016
2016
Cited alongside, same era.
E. Kiperwasser and Y. Goldberg, “Simple and accurate dependency parsing using bidirectional lstm feature representations,” TACL , vol. 4, pp. 313–327, 2016
2016
Cited alongside, same era.
A. P. Parikh, O. Täckström, D. Das, and J. Uszkoreit, “A decomposable attention model for natural language inference,” in Proceedings of EMNLP , 2016
2016
Cited alongside, same era.
Y. Niu, R. Xie, Z. Liu, and M. Sun, “Improved word representation learning with sememes,” in Proceedings of ACL , 2017
2017
Cited alongside, same era.
B. Yang and T. Mitchell, “Leveraging knowledge bases in lstms for improving machine reading,” in Proceedings of ACL , 2017
2017
Cited alongside, same era.

Then

T. Young, E. Cambria, I. Chaturvedi, H. Zhou, S. Biswas, and M. Huang, “Augmenting end-to-end dialogue systems with commonsense knowledge,” in Proceedings of AAAI , 2018
2018
Later among the works it cites.
Y. Ma, H. Peng, and E. Cambria, “Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive lstm,” in Proceedings of AAAI , 2018
2018
Later among the works it cites.
M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proceedings of the NAACL-HLT , 2018
2018
Later among the works it cites.
A. Conneau and D. Kiela, “Senteval: An evaluation toolkit for universal sentence representations,” in Proceedings of LREC , 2018
2018
Later among the works it cites.
M. Alzantot, Y. Sharma, A. Elgohary, B.-J. Ho, M. Srivastava, and K.-W. Chang, “Generating natural language adversarial examples,” in Proceedings of EMNLP , 2018
2018
Later among the works it cites.
H. Jin, H. Zhu, Z. Liu, R. Xie, M. Sun, F. Lin, and L. Lin, “Incorporating chinese characters of words for lexical sememe prediction,” in Proceedings of ACL , 2018
2018
Later among the works it cites.
F. Qi, Y. Lin, M. Sun, H. Zhu, R. Xie, and Z. Liu, “Cross-lingual lexical sememe prediction,” in Proceedings of EMNLP , 2018
2018
Later among the works it cites.
L. He, K. Lee, O. Levy, and L. Zettlemoyer, “Jointly predicting predicates and arguments in neural semantic role labeling,” in Proceedings of ACL , 2018
2018
Later among the works it cites.
F. Qi, C. Yang, Z. Liu, Q. Dong, M. Sun, and Z. Dong, “OpenHowNet: An open sememe-based lexical knowledge base,” arXiv preprint arXiv:1901.09957 , 2019
Original
2019
Closest in time.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT , 2019
2019
Closest in time.
W. E. Zhang, Q. Z. Sheng, A. Alhazmi, and L. Chenliang, “Adversarial attacks on deep learning models in natural language processing: A survey,” arXiv preprint arXiv:1901.06796 , 2019
Original
2019
Closest in time.
W. Wang, L. Wang, B. Tang, R. Wang, and A. Ye, “A survey: Towards a robust deep neural network in text domain,” arXiv preprint arXiv:1902.07285 , 2019
Original
2019
Closest in time.
F. Qi, J. Huang, C. Yang, Z. Liu, X. Chen, Q. Liu, and M. Sun, “Modeling semantic compositionality with sememe knowledge,” in Proceedings of ACL , 2019
2019
Closest in time.
Y. Zang, F. Qi, C. Yang, Z. Liu, M. Zhang, Q. Liu, and M. Sun, “Word-level textual adversarial attacking as combinatorial optimization,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020, pp. 6066–6080
2020
Closest in time.
L. Zhang, F. Qi, Z. Liu, Y. Wang, Q. Liu, and M. Sun, “Multi-channel reverse dictionary model,” in Proceedings of AAAI , 2020
2020
Closest in time.
F. Qi, L. Chang, M. Sun, O. Sicong, and Z. Liu, “Towards building a multilingual sememe knowledge base: Predicting sememes for BabelNet synsets,” in Proceedings of AAAI , 2020
2020
Closest in time.
A. Graves, S. Fernández, and J. Schmidhuber, “Bidirectional lstm networks for improved phoneme classification and recognition,” in Proceedings of International Conference on Artificial Neural Networks , 2005
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…