Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Improving language models by retrieving from trillions of tokens

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022 · 2022

Earlier work this paper cites.

Glm: General language model pretraining with autoregressive blank infilling

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022 · 2022

Earlier work this paper cites.

Train short, test long: Attention with linear biases enables input length extrapolation

Ofir Press, Noah Smith, and Mike Lewis. 2022 · 2022

Earlier work this paper cites.

Qwen technical report

Original

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023 · 2023

Earlier work this paper cites.

Ntk-aware scaled rope

bloc97. 2023 · 2023

Earlier work this paper cites.

Extending context window of large language models via positional interpolation

Original

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023 · 2023

Earlier work this paper cites.

Time travel in llms: Tracing data contamination in large language models

Original

Shahriar Golchin and Mihai Surdeanu. 2023 · 2023

Earlier work this paper cites.

Needle in a haystack - pressure testing llms

Greg Kamradt. 2023 · 2023

Earlier work this paper cites.

Gpt-4 technical report

Original

OpenAI. 2023 · 2023

Earlier work this paper cites.

Giraffe: Adventures in expanding context lengths in llms

Original

Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. 2023 · 2023

Earlier work this paper cites.

Parallel context windows for large language models

Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023 · 2023

Earlier work this paper cites.

Code llama: Open foundation models for code

Original

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023 · 2023

Cited alongside, same era.

A length-extrapolatable transformer

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2023 · 2023

Cited alongside, same era.

Efficient streaming language models with attention sinks

Original

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023 · 2023

Cited alongside, same era.

Wider and deeper llm networks are fairer llm evaluators

Original

Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023 · 2023

Cited alongside, same era.

L-eval: Instituting standardized evaluation for long context language models

Clongeval: A chinese benchmark for evaluating long-context large language models

Original

Zexuan Qiu, Jingjing Li, Shijue Huang, Wanjun Zhong, and Irwin King. 2024 · 2024

Closest in time.

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Original

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024 · 2024

Closest in time.

REPLUG: Retrieval-augmented black-box language models

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024 · 2024

Closest in time.

Counting-stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models

Original

Mingyang Song, Mao Zheng, and Xuan Luo. 2024 · 2024

Closest in time.

Roformer: Enhanced transformer with rotary position embedding