Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

Language Models are Few-Shot Learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

Randomized Positional Encodings Boost Length Generalization of Transformers

Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. 2023 · 1903

Earlier work this paper cites.

Syntactic structures

Noam Chomsky. 1957 · 1957

Earlier work this paper cites.

Universal grammar

RICHARD MONTAGUE. 1970 · 1970

Earlier work this paper cites.

Layer Normalization

Original

Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016 · 2016

Earlier work this paper cites.

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 · 2016

Earlier work this paper cites.

Attention is All you Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Arthur Jacot, Franck Gabriel, and Clement Hongler. 2018 · 2018

Earlier work this paper cites.

Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks

Brenden Lake and Marco Baroni. 2018 · 2018

Earlier work this paper cites.

Extrapolation in NLP

Jeff Mitchell, Pontus Stenetorp, Pasquale Minervini, and Sebastian Riedel. 2018 · 2018

Earlier work this paper cites.

Self-Attention with Relative Position Representations

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018 · 2018

Earlier work this paper cites.

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019 · 2019

Earlier work this paper cites.

On the Relation between Position Information and Sentence Length in Neural Machine Translation

Masato Neishi and Naoki Yoshinaga. 2019 · 2019

Earlier work this paper cites.

Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel

Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019 · 2019

Earlier work this paper cites.

Encoding word order in complex embeddings

Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. 2019 · 2019

Earlier work this paper cites.

Are Transformers universal approximators of sequence-to-sequence functions?

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. 2019 · 2019

Earlier work this paper cites.

Location Attention for Extrapolation to Longer Sequences

Yann Dubois, Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. 2020 · 2020

Earlier work this paper cites.

Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2020

Earlier work this paper cites.

Improve Transformer Models with Better Relative Position Embeddings

Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. 2020 · 2020

Earlier work this paper cites.

Rethinking Positional Encoding in Language Pre-training

Guolin Ke, Di He, and Tie-Yan Liu. 2020 · 2020

Earlier work this paper cites.

Learning to Encode Position for Transformer with Continuous Dynamical Model

Xuanqing Liu, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. 2020 · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020

Earlier work this paper cites.

On Position Embeddings in BERT

Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen. 2020 · 2020

Earlier work this paper cites.

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020 · 2020

Earlier work this paper cites.

SHAPE: Shifted Absolute Position Embedding for Transformers

Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. 2021 · 2021

Earlier work this paper cites.

CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings

Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozhnikov. 2021 · 2021

Earlier work this paper cites.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Ofir Press, Noah Smith, and Mike Lewis. 2021 · 2021

Earlier work this paper cites.

Scale Efficiently: Insights from Pretraining and Finetuning Transformers

Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2021 · 2021

Earlier work this paper cites.

The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models

Ulme Wennberg and Gustav Eje Henter. 2021 · 2021

Earlier work this paper cites.

Exploring Length Generalization in Large Language Models

Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022 · 2022

Earlier work this paper cites.

Recurrent Memory Transformer

Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022 · 2022

Earlier work this paper cites.

KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation

Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge, and Alexander Rudnicky. 2022 · 2022

Cited alongside, same era.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022 · 2022

Cited alongside, same era.

Position Information in Transformers: An Overview

Philipp Dufter, Martin Schmitt, and Hinrich Schütze. 2022 · 2022

Cited alongside, same era.

Efficiently Modeling Long Sequences with Structured State Spaces

Original

Albert Gu, Karan Goel, and Christopher Ré. 2022 · 2022

Cited alongside, same era.

Transformer Language Models without Positional Encodings Still Learn Positional Information

Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022 · 2022

Cited alongside, same era.

ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023 · 2023

Closest in time.

Structured Packing in LLM Training Improves Long Context Utilization

Original

Konrad Staniszewski, Szymon Tworkowski, Sebastian Jaszczur, Henryk Michalewski, Łukasz Kuciński, and Piotr Miłoś. 2023 · 2023

Closest in time.

A Length-Extrapolatable Transformer

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2023 · 2023

Closest in time.

Effective Long-Context Scaling of Foundation Models

Original

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. 2023 · 2023

Closest in time.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023 · 2023