Lossless KV Cache Compression to 2%

Socialiqa: Commonsense reasoning about social interactions

Original

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019 · 1904

Earlier work this paper cites.

Boolq: Exploring the surprising difficulty of natural yes/no questions

Original

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019 · 1905

Earlier work this paper cites.

Hellaswag: Can a machine really finish your sentence?

Original

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 1905

Earlier work this paper cites.

Fast transformer decoding: One write-head is all you need

Original

Noam Shazeer. 2019 · 1911

Earlier work this paper cites.

Glu variants improve transformer

Original

Noam Shazeer. 2020 · 2002

Earlier work this paper cites.

Clue: A chinese language understanding evaluation benchmark

Original

Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020 · 2004

Earlier work this paper cites.

Gshard: Scaling giant models with conditional computation and automatic sharding

Original

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020 · 2006

Earlier work this paper cites.

Easyquant: Post-training quantization via scale optimization

Original

Di Wu, Qi Tang, Yongle Zhao, Ming Zhang, Ying Fu, and Debing Zhang. 2020 · 2006

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009

Earlier work this paper cites.

Ocnli: Original chinese natural language inference

Original

Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence S Moss. 2020 · 2010

Earlier work this paper cites.

Estimating or propagating gradients through stochastic neurons for conditional computation

Original

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013 · 2013

Earlier work this paper cites.

Thuctc: An efficient chinese text classifier

Maosong Sun, Jingyang Li, Zhipeng Guo, Z Yu, Y Zheng, X Si, and Z Liu. 2016 · 2016

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Scalable methods for 8-bit training of neural networks

Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. 2018 · 2018

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Original

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 · 2018

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019 · 2019

Earlier work this paper cites.

Root mean square layer normalization

Biao Zhang and Rico Sennrich. 2019 · 2019

Earlier work this paper cites.

Piqa: Reasoning about physical commonsense in natural language

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020 · 2020

Earlier work this paper cites.

Investigating prior knowledge for challenging chinese machine reading comprehension

Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020 · 2020

Earlier work this paper cites.

A white paper on neural network quantization

Original

Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. 2021 · 2021

Cited alongside, same era.

Winogrande: An adversarial winograd schema challenge at scale

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021 · 2021

Cited alongside, same era.

Enhanced transformer with rotary position embedding., 2021

J Su, Y Lu, S Pan, A Murtadha, B Wen, and Y Liu Roformer. 2023 · 2021

Cited alongside, same era.

Fewclue: A chinese few-shot learning evaluation benchmark

Original

Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Huilin Xu, Hu Yuan, Guoao Wei, Xiang Pan, Xin Tian, Libo Qin, et al. 2021 · 2021

Cited alongside, same era.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

William Fedus, Barret Zoph, and Noam Shazeer. 2022 · 2022

Large language models for information retrieval: A survey

Original

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023 · 2023

Later among the works it cites.

Llama 3 model card

AI@Meta. 2024 · 2024

Closest in time.

Reducing transformer key-value cache size with cross-layer attention

Original

William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. 2024 · 2024

Closest in time.

Learning to compress prompt in natural language formats

Original

Yu-Neng Chuang, Tianwei Xing, Chia-Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. 2024 · 2024

Closest in time.

Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

Original

DeepSeek-AI. 2024 · 2024