What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding

Transformer-xl: Attentive language models beyond a fixed-length context

Original

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019 · 1901

Earlier work this paper cites.

fairseq: A fast, extensible toolkit for sequence modeling

Original

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019 · 1904

Earlier work this paper cites.

Roberta: A robustly optimized bert pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907

Earlier work this paper cites.

Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

Original

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 1910

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Original

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 1910

Earlier work this paper cites.

Encoding word order in complex embeddings

Original

Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. 2019 · 1912

Earlier work this paper cites.

Backpropagation through time: what it does and how to do it

Paul J Werbos. 1990 · 1990

Earlier work this paper cites.

A new algorithm for data compression

Philip Gage. 1994 · 1994

Earlier work this paper cites.

Convolutional networks for images, speech, and time series

Yann LeCun, Yoshua Bengio, et al. 1995 · 1995

Earlier work this paper cites.

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997

Cited alongside, same era.

Learning meronyms from biomedical text

Angus Roberts. 2005 · 2005

Cited alongside, same era.

Neural machine translation by jointly learning to align and translate

Original

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014

Cited alongside, same era.

Empirical evaluation of gated recurrent neural networks on sequence modeling

Original

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014 · 2014

Cited alongside, same era.

Sequence to sequence learning with neural networks

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014 · 2014

Cited alongside, same era.

Neural machine translation of rare words with subword units

Convolutional sequence to sequence learning

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017 · 2017

Later among the works it cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Later among the works it cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Later among the works it cites.

Deep contextualized word representations

Original

Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018

Later among the works it cites.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018

Later among the works it cites.