RefreshKV: Updating Small KV Cache During Long-form Generation

Generating long sequences with sparse transformers

Original

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019 · 1904

Earlier work this paper cites.

Longformer: The long-document transformer

Original

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020 · 2004

Earlier work this paper cites.

Big bird: Transformers for longer sequences

Original

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020 · 2007

Earlier work this paper cites.

8-bit optimizers via block-wise quantization

Original

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2021 · 2021

Earlier work this paper cites.

Efficient attentions for long document summarization

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021 · 2021

Earlier work this paper cites.

QMSum: A new benchmark for query-based multi-domain meeting summarization

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021 · 2021

Earlier work this paper cites.

Smoothquant: Accurate and efficient post-training quantization for large language models

Original

Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. 2022 · 2022

Earlier work this paper cites.

Gqa: Training generalized multi-query transformer models from multi-head checkpoints

Original

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr’on, and Sumit K. Sanghai. 2023 · 2023

Earlier work this paper cites.

L-eval: Instituting standardized evaluation for long context language models

Original

Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023 · 2023

Earlier work this paper cites.

Longbench: A bilingual, multitask benchmark for long context understanding

Original

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023 · 2023

Earlier work this paper cites.

Unlimiformer: Long-range transformers with unlimited length input

Original

Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. 2023 · 2023

Earlier work this paper cites.

Needle in a haystack - pressure testing llms, commercially usable llms

Gregory Kamradt. 2023 · 2023

Cited alongside, same era.

Introducing mpt-7b: A new standard for open-source, commercially usable llms

NLP Team MosaicML. 2023 · 2023

Cited alongside, same era.

Redpajama: an open dataset for training large language models

Together. 2023 · 2023

Cited alongside, same era.

Efficient streaming language models with attention sinks

Original

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023 · 2023

Cited alongside, same era.

Pytorch fsdp: Experiences on scaling fully sharded data parallel

Yanli Zhao, Andrew Gu, Rohan Varma, Liangchen Luo, Chien chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, and Shen Li. 2023 · 2023

Cited alongside, same era.

Kvquant: Towards 10 million context length llm inference with kv cache quantization

Original

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024 · 2024

Closest in time.

Ruler: What’s the real context size of your long-context language models?

Original

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024 · 2024

Closest in time.

Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

Original

Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024 · 2024

Closest in time.

One thousand and one pairs: A "novel" challenge for long-context language models

Original

Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024 · 2024

Closest in time.