2020

ETC: Encoding Long and Structured Inputs in Transformers

Ainslie, Joshua, Ontanon, Santiago, Alberti, Chris et al.

Understand

Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks.

In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs.
To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens.
We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pre-training objective allows ETC to encode structured inputs.

Built on

A bert baseline for the natural questions
Original
Chris Alberti, Kenton Lee, and Michael Collins. 2019 · 1901
Earlier work this paper cites.
Transformer-xl: Attentive language models beyond a fixed-length context
Original
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019 · 1901
Earlier work this paper cites.
Star-transformer
Original
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019 · 1902
Earlier work this paper cites.
Selective attention for context-aware neural machine translation
Original
Sameen Maruf, André FT Martins, and Gholamreza Haffari. 2019 · 1903
Earlier work this paper cites.
Generating long sequences with sparse transformers
Original
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019 · 1904
Earlier work this paper cites.
Generating logical forms from graph representations of text and entities
Original
Peter Shaw, Philip Massey, Angelica Chen, Francesco Piccinno, and Yasemin Altun. 2019 · 1905
Earlier work this paper cites.
Adaptive attention span in transformers
Original
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019 · 1905
Earlier work this paper cites.
Hibert: Document level pre-training of hierarchical bidirectional transformers for document summarization
Original
Xingxing Zhang, Furu Wei, and Ming Zhou. 2019 · 1905
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Bert for coreference resolution: Baselines and analysis
Original
Mandar Joshi, Omer Levy, Daniel S Weld, and Luke Zettlemoyer. 2019 · 1908
Earlier work this paper cites.
Adaptively sparse transformers
Original
Gonçalo M Correia, Vlad Niculae, and André FT Martins. 2019 · 1909
Earlier work this paper cites.
Albert: A lite bert for self-supervised learning of language representations
Original
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019 · 1909
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 1910
Earlier work this paper cites.
Compressive transformers for long-range sequence modelling
Original
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. 2019 · 1911
Earlier work this paper cites.
Open domain web keyphrase extraction beyond language modeling
Original
Lee Xiong, Chuan Hu, Chenyan Xiong, Daniel Campos, and Arnold Overwijk. 2019 · 1911
Earlier work this paper cites.

Similar

Bp-transformer: Modelling long-range context via binary partitioning
Original
Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019 · 1911
Cited alongside, same era.
Reformer: The efficient transformer
Original
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020 · 2001
Cited alongside, same era.
Efficient content-based sparse attention with routing transformers
Original
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020 · 2003
Cited alongside, same era.
Longformer: The long-document transformer
Original
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020 · 2004
Cited alongside, same era.
Rikinet: Reading wikipedia pages for natural question answering
Original
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020 · 2004
Cited alongside, same era.
Joint keyphrase chunking and salience ranking with bert
Original
Si Sun, Chenyan Xiong, Zhenghao Liu, Zhiyuan Liu, and Jie Bao. 2020 · 2004
Cited alongside, same era.
Big bird: Transformers for longer sequences
Original
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020 · 2007
Cited alongside, same era.
The graph neural network model
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008 · 2008
Cited alongside, same era.
Practical and optimal lsh for angular distance
Alexandr Andoni, Piotr Indyk, Thijs Laarhoven, Ilya Razenshteyn, and Ludwig Schmidt. 2015 · 2015
Cited alongside, same era.
Squad: 100,000+ questions for machine comprehension of text
Original
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016
Cited alongside, same era.
Hierarchical attention networks for document classification
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016 · 2016
Cited alongside, same era.
The reversible residual network: Backpropagation without storing activations
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. 2017 · 2017
Cited alongside, same era.

Then

Music transformer: Generating music with long-term structure
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. 2018 · 2018
Later among the works it cites.
Generating wikipedia by summarizing long sequences
Original
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018 · 2018
Later among the works it cites.
Document-level neural machine translation with hierarchical attention networks
Original
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018 · 2018
Later among the works it cites.
Representation learning with contrastive predictive coding
Original
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018 · 2018
Later among the works it cites.
Self-attention with relative position representations
Original
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018 · 2018
Later among the works it cites.
Adafactor: Adaptive learning rates with sublinear memory cost
Original
Noam Shazeer and Mitchell Stern. 2018 · 2018
Later among the works it cites.
Bi-directional block self-attention for fast and memory-efficient sequence modeling
Original
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. 2018 · 2018
Later among the works it cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding
Original
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018 · 2018
Later among the works it cites.
Constructing datasets for multi-hop reading comprehension across documents
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018 · 2018
Later among the works it cites.
Hotpotqa: A dataset for diverse, explainable multi-hop question answering
Original
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018 · 2018
Later among the works it cites.
Natural questions: a benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019 · 2019
Later among the works it cites.
Xlnet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019 · 2019
Later among the works it cites.
Large batch optimization for deep learning: Training bert in 76 minutes
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019 · 2019
Later among the works it cites.
Leveraging pre-trained checkpoints for sequence generation tasks
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…