Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

End-to-end memory networks

Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus · 2015

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt · 2020

Earlier work this paper cites.

GLU variants improve transformer

Original

Noam Shazeer · 2020

Earlier work this paper cites.

Editing factual knowledge in language models

Nicola De Cao, Wilker Aziz, and Ivan Titov · 2021

Earlier work this paper cites.

Transformer feed-forward layers are key-value memories

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy · 2021

Earlier work this paper cites.

Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX

Ben Wang · 2021

Cited alongside, same era.

Ext5: Towards extreme multi-task scaling for transfer learning

Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler · 2022

Cited alongside, same era.

Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space

Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg · 2022

Cited alongside, same era.

Finetuned language models are zero-shot learners

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le · 2022

Cited alongside, same era.

Transformer-patcher: One mistake worth one neuron

Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong · 2023

Cited alongside, same era.

Mass-editing memory in a transformer

Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau · 2023

Later among the works it cites.

Stanford alpaca: An instruction-following llama model

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto · 2023

Later among the works it cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023

Later among the works it cites.

Knowledge editing for large language models: A survey

Original

Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, et al · 2023

Later among the works it cites.