UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective

Fast transformer decoding: One write-head is all you need

Original

Noam Shazeer. 2019 · 1911

Earlier work this paper cites.

Who belongs in the family?

Robert L Thorndike. 1953 · 1953

Earlier work this paper cites.

The application of electronic computers to factor analysis

Henry F Kaiser. 1960 · 1960

Earlier work this paper cites.

On measures of entropy and information

Alfréd Rényi. 1961 · 1961

Earlier work this paper cites.

Principal components analysis (pca)

Andrzej Maćkiewicz and Waldemar Ratajczak. 1993 · 1993

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

A fast learning algorithm for deep belief nets

Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. 2006 · 2006

Earlier work this paper cites.

Scaling learning algorithms towards AI

Yoshua Bengio and Yann LeCun. 2007 · 2007

Earlier work this paper cites.

The effective rank: A measure of effective dimensionality

Olivier Roy and Martin Vetterli. 2007 · 2007

Earlier work this paper cites.

Pearson correlation coefficient

Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009 · 2009

Earlier work this paper cites.

Mathematische grundlagen der quantenmechanik , volume 38

John Von Neumann. 2013 · 2013

Earlier work this paper cites.

Measures of entropy from data using infinitely divisible kernels

Luis Gonzalo Sanchez Giraldo, Murali Rao, and Jose C Principe. 2014 · 2014

Earlier work this paper cites.

Deep learning , volume 1

Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016 · 2016

Earlier work this paper cites.

The wikitext long term dependency language modeling dataset

Stephen Merity. 2016 · 2016

Earlier work this paper cites.

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Original

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Original

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Earlier work this paper cites.

Rank diminishing in deep neural networks

Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. 2022 · 2022

Earlier work this paper cites.

Language model compression with weighted low-rank factorization

Original

Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Earlier work this paper cites.

Gqa: Training generalized multi-query transformer models from multi-head checkpoints

Original

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023 · 2023

Cited alongside, same era.

Longbench: A bilingual, multitask benchmark for long context understanding

Original

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023 · 2023

Cited alongside, same era.

Flashattention-2: Faster attention with better parallelism and work partitioning

Original

Tri Dao. 2023 · 2023

Cited alongside, same era.

Language modeling is compression

Original

Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. 2023 · 2023

Cited alongside, same era.

Llama 3: A family of large language models

Meta AI. 2024 · 2024

Closest in time.

Reducing transformer key-value cache size with cross-layer attention

Original

William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. 2024 · 2024

Closest in time.

Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

Original

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. 2024 · 2024

Closest in time.

Kvquant: Towards 10 million context length llm inference with kv cache quantization

Original

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024 · 2024

Closest in time.

Ruler: What’s the real context size of your long-context language models?