2020

Tree-structured Attention with Hierarchical Accumulation

Nguyen, Xuan-Phi, Joty, Shafiq, Hoi, Steven C. H. et al.

Understand

Incorporating hierarchical structures like constituency trees has been shown to be effective for various natural language processing (NLP) tasks.

However, it is evident that state-of-the-art (SOTA) sequence-based models like the Transformer struggle to encode such structures inherently.
On the other hand, dedicated models like the Tree-LSTM, while explicitly modeling hierarchical structures, do not perform as efficiently as the Transformer.
In this paper, we attempt to bridge this gap with "Hierarchical Accumulation" to encode parse tree structures into self-attention at constant time complexity.

Built on

Recursive deep models for semantic compositionality over a sentiment treebank
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts · 2013
Earlier work this paper cites.
The Stanford CoreNLP natural language processing toolkit
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky · 2014
Earlier work this paper cites.
Improved semantic representations from tree-structured long short-term memory networks
Kai Sheng Tai, Richard Socher, and Christopher D. Manning · 2015
Earlier work this paper cites.
Tree-to-sequence attentional neural machine translation
Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka · 2016
Earlier work this paper cites.
Assessing the ability of lstms to learn syntax-sensitive dependencies
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg · 2016
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2016
Earlier work this paper cites.
Tree-structured decoding with doubly-recurrent neural networks
David Alvarez-Melis and Tommi S Jaakkola · 2017
Earlier work this paper cites.
Convolutional Sequence to Sequence Learning
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.

Similar

Towards bidirectional hierarchical representations for attention-based neural machine translation
Baosong Yang, Derek F. Wong, Tong Xiao, Lidia S. Chao, and Jingbo Zhu · 2017
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
Top-down tree structured decoding with syntactic connections for neural machine translation and parsing
Jetic Gū, Hassan S. Shavarani, and Anoop Sarkar · 2018
Cited alongside, same era.
Scaling neural machine translation
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli · 2018
Cited alongside, same era.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever · 2018
Cited alongside, same era.
Straight to the tree: Constituency parsing with neural syntactic distance
Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron Courville, and Yoshua Bengio · 2018
Cited alongside, same era.
On tree-based neural sentence modeling
Haoyue Shi, Hao Zhou, Jiaze Chen, and Lei Li · 2018
Cited alongside, same era.
Linguistically-informed self-attention for semantic role labeling
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum · 2018
Cited alongside, same era.
Multi-granularity self-attention for neural machine translation
Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu · 2019
Cited alongside, same era.
über sinn und bedeutung
Gottlob Frege
Cited in the paper.

Then

Tree-transformer: A transformer-based method for correction of tree-structured data
Original
Jacob Harer, Chris Reale, and Peter Chin · 2019
Later among the works it cites.
A structural probe for finding syntax in word representations
John Hewitt and Christopher D Manning · 2019
Later among the works it cites.
Unsupervised recurrent neural network grammars
Original
Yoon Kim, Alexander M Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gábor Melis · 2019
Later among the works it cites.
Cross-lingual language model pretraining
Original
Guillaume Lample and Alexis Conneau · 2019
Later among the works it cites.
Ordered neurons: Integrating tree structures into recurrent neural networks
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville · 2019
Later among the works it cites.
Pay less attention with lightweight and dynamic convolutions
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli · 2019
Later among the works it cites.
Xlnet: Generalized autoregressive pretraining for language understanding
Original
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le · 2019
Later among the works it cites.
Multitree transformer: Integrating tree structures into self-attention
Hung-Yi Lee Yaushian Wang and Yun-Nung Chen · 2019
Later among the works it cites.
Self-attention with relative position representations
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani · 2074
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…