Fetching the paper…

Reading the bibliography…

2024

UniMem: Towards a Unified View of Long-Context Large Language Models

Fang, Junjie, Tang, Likai, Bi, Hongzhe et al.

Original
PDF
Post on X

Understand

Long-context processing is a critical ability that constrains the applicability of large language models (LLMs).

Although there exist various methods devoted to enhancing the long-context processing ability of LLMs, they are developed in an isolated manner and lack systematic analysis and integration of their strengths, hindering further developments.
In this paper, we introduce UniMem, a Unified framework that reformulates existing long-context methods from the view of Memory augmentation of LLMs.
Distinguished by its four core dimensions-Memory Management, Memory Writing, Memory Reading, and Memory Injection, UniMem empowers researchers to conduct systematic exploration of long-context methods.

UniMem: Towards a Unified View of Long-Context Large Language Models · Around

Built on

Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Transformer-xl: Attentive language models beyond a fixed-length context
Original
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov · 2019
Earlier work this paper cites.
Generalization through memorization: Nearest neighbor language models
Original
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis · 2019
Earlier work this paper cites.
Compressive transformers for long-range sequence modelling
Original
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap · 2019
Earlier work this paper cites.
Longformer: The long-document transformer
Original
Iz Beltagy, Matthew E Peters, and Arman Cohan · 2020
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Earlier work this paper cites.
Reformer: The efficient transformer
Original
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya · 2020
Earlier work this paper cites.
Big bird: Transformers for longer sequences
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al · 2020
Earlier work this paper cites.
Training language models with memory augmentation
Original
Zexuan Zhong, Tao Lei, and Danqi Chen · 2020
Earlier work this paper cites.
On the opportunities and risks of foundation models
Original
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al · 2021
Earlier work this paper cites.
Efficiently modeling long sequences with structured state spaces
Original
Albert Gu, Karan Goel, and Christopher Ré · 2021
Earlier work this paper cites.

Similar

Longt5: Efficient text-to-text transformer for long sequences
Original
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang · 2021
Cited alongside, same era.
Efficient content-based sparse attention with routing transformers
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier · 2021
Cited alongside, same era.
Do long-range language models actually use long-range context?
Original
Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer · 2021
Cited alongside, same era.
Poolingformer: Long document modeling with pooling attention
Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen · 2021
Cited alongside, same era.
Recurrent memory transformer
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev · 2022
Cited alongside, same era.

Then

Longnet: Scaling transformers to 1,000,000,000 tokens
Original
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei · 2023
Later among the works it cites.
Retrieval-augmented generation for large language models: A survey
Original
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang · 2023
Later among the works it cites.
Mamba: Linear-time sequence modeling with selective state spaces
Original
Albert Gu and Tri Dao · 2023
Later among the works it cites.
Yarn: Efficient context window extension of large language models
Original
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole · 2023
Later among the works it cites.
Retentive network: A successor to transformer for large language models

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Competition-level code generation with alphacode

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al · 2022

Cited alongside, same era.

OpenAI: Introducing ChatGPT, 2022

OpenAI · 2022

Cited alongside, same era.

Memorizing transformers

Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy · 2022

Cited alongside, same era.

Unlimiformer: Long-range transformers with unlimited length input

Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R Gormley · 2023

Cited alongside, same era.

Adapting language models to compress contexts

Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen · 2023

Cited alongside, same era.

Redpajama: an open dataset for training large language models, 2023

Together Computer · 2023

Cited alongside, same era.

Extending context window of large language models via positional interpolation

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian

Cited in the paper.

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei · 2023

Later among the works it cites.

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al · 2023

Later among the works it cites.

Focused transformer: Contrastive training for context scaling

Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś · 2023

Later among the works it cites.

Efficient streaming language models with attention sinks

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis · 2023

Later among the works it cites.

Effective long-context scaling of foundation models

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al · 2023

Later among the works it cites.

Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica · 2023

Later among the works it cites.

Tinyllama: An open-source small language model, 2024

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu · 2024

Closest in time.