Transformer Language Models without Positional Encodings Still Learn Positional Information

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

Roberta: A robustly optimized BERT pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907

Earlier work this paper cites.

A method for unconstrained convex minimization problem with the rate of convergence o ⁡ ( 1 / k 2 ) o(1/k^{2})

Yurii Nesterov. 1983 · 1983

Earlier work this paper cites.

Neural machine translation by jointly learning to align and translate

Original

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015

Earlier work this paper cites.

Adam: A method for stochastic optimization

Original

Diederik P. Kingma and Jimmy Ba. 2015 · 2015

Earlier work this paper cites.

End-to-end memory networks

Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015 · 2015

Earlier work this paper cites.

Convolutional sequence to sequence learning

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017 · 2017

Earlier work this paper cites.

Pointer sentinel mixture models

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Cited alongside, same era.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018

Cited alongside, same era.

Self-attention with relative position representations

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018 · 2018

Cited alongside, same era.

Adaptive input representations for neural language modeling

Alexei Baevski and Michael Auli. 2019 · 2019

Cited alongside, same era.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Cited alongside, same era.

Language modeling with deep transformers

Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019 · 2019

The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives

Elena Voita, Rico Sennrich, and Ivan Titov. 2019 · 2019

Later among the works it cites.

The Pile: An 800gb dataset of diverse text for language modeling

Original

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020 · 2020

Later among the works it cites.

Improving transformer models by reordering their sublayers

Ofir Press, Noah A. Smith, and Omer Levy. 2020 · 2020

Later among the works it cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020

Later among the works it cites.

Masked language modeling and the distributional hypothesis: Order word matters pre-training for little

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021 · 2021