TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection

Megatron-lm: Training multi-billion parameter language models using model parallelism

Original

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020 · 1909

Earlier work this paper cites.

Neural tangent kernel: Convergence and generalization in neural networks

Arthur Jacot, Franck Gabriel, and Clément Hongler. 2018 · 2018

Earlier work this paper cites.

Transformer-XL: Language modeling with longer-term dependency

Zihang Dai*, Zhilin Yang*, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019 · 2019

Earlier work this paper cites.

Pytorch: An imperative style, high-performance deep learning library

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 2019 · 2019

Earlier work this paper cites.

Triton: an intermediate language and compiler for tiled neural network computations

Philippe Tillet, H. T. Kung, and David Cox. 2019 · 2019

Earlier work this paper cites.

Multi-passage bert: A globally normalized bert model for open-domain question answering

Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019 · 2019

Earlier work this paper cites.

Compressive transformers for long-range sequence modelling

Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020 · 2020

Earlier work this paper cites.

Big bird: Transformers for longer sequences

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020 · 2020

Earlier work this paper cites.

Recurrent memory transformer

Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022 · 2022

Earlier work this paper cites.

Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

bloc97. 2023 · 2023

Earlier work this paper cites.

Extending context window of large language models via positional interpolation

Original

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023 · 2023

Earlier work this paper cites.

Dynamically scaled rope further increases performance of long context llama with zero fine-tuning

emozilla. 2023 · 2023

Earlier work this paper cites.

Mistral 7b

Original

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023 · 2023

Earlier work this paper cites.

Efficient memory management for large language model serving with pagedattention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023 · 2023

Earlier work this paper cites.

Giraffe: Adventures in expanding context lengths in llms

Original

Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. 2023 · 2023

Earlier work this paper cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, and 1 others. 2023 · 2023

Cited alongside, same era.

Length extrapolation of transformers: A survey from the perspective of positional encoding

Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, and Ting Liu. 2023 · 2023

Cited alongside, same era.

Yi: Open foundation models by 01.ai

Original

01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, and 13 others. 2024 · 2024

Cited alongside, same era.

LongBench: A bilingual, multitask benchmark for long context understanding

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 · 2024

Cited alongside, same era.

Snapkv: Llm knows what you are looking for before generation

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024 · 2024

Closest in time.

Leave no context behind: Efficient infinite context transformers with infini-attention

Original

Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. 2024 · 2024

Closest in time.

Introducing OpenAI o1

OpenAI · 2024

Closest in time.

Transformers are multi-state RNNs

Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. 2024 · 2024

Closest in time.

YaRN: Efficient context window extension of large language models

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024 · 2024

Closest in time.

Sparq attention: Bandwidth-efficient LLM inference

Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. 2024 · 2024