A Survey on Model Compression for Large Language Models

Democratizing reasoning ability: Tailored learning from large language model

Zhaoyang Wang, Shaohan Huang, Yuxuan Liu, Jiahai Wang, Minghui Song, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2023f · 1966

Earlier work this paper cites.

Optimal brain damage

Yann LeCun, John S. Denker, and Sara A. Solla. 1989 · 1989

Earlier work this paper cites.

Building a large annotated corpus of English: The Penn Treebank

Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993 · 1993

Earlier work this paper cites.

Quantization

R.M. Gray and D.L. Neuhoff. 1998 · 1998

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

Weighted low-rank approximations

Nathan Srebro and Tommi S. Jaakkola. 2003 · 2003

Earlier work this paper cites.

Piqa: An algebra for querying protein data sets

Sandeep Tata and Jignesh M. Patel. 2003 · 2003

Earlier work this paper cites.

Roofline: an insightful visual performance model for multicore architectures

Samuel Williams, Andrew Waterman, and David A. Patterson. 2009 · 2009

Earlier work this paper cites.

Learning both weights and connections for efficient neural network

Song Han, Jeff Pool, John Tran, and William Dally. 2015 · 2015

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015 · 2015

Earlier work this paper cites.

Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding

Original

Song Han, Huizi Mao, and William J. Dally. 2016 · 2016

Earlier work this paper cites.

The LAMBADA dataset: Word prediction requiring a broad discourse context

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016 · 2016

Earlier work this paper cites.

Xnor-net: Imagenet classification using binary convolutional neural networks

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. 2016 · 2016

Earlier work this paper cites.

Learning structured sparsity in deep neural networks

Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2016 · 2016

Earlier work this paper cites.

Model-agnostic meta-learning for fast adaptation of deep networks

Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017 · 2017

Earlier work this paper cites.

Pointer sentinel mixture models

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017 · 2017

Earlier work this paper cites.

Neural architecture search with reinforcement learning

Barret Zoph and Quoc V. Le. 2017 · 2017

Earlier work this paper cites.

Can a suit of armor conduct electricity? A new dataset for open book question answering

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 · 2018

Earlier work this paper cites.

The lottery ticket hypothesis: Finding sparse, trainable neural networks

Jonathan Frankle and Michael Carbin. 2019 · 2019

Earlier work this paper cites.

Importance estimation for neural network pruning

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Earlier work this paper cites.

Learning implicitly recurrent cnns through parameter sharing

Pedro Savarese and Michael Maire. 2019 · 2019

Earlier work this paper cites.

CommonsenseQA: A question answering challenge targeting commonsense knowledge

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019 · 2019

Earlier work this paper cites.

Language models are few-shot learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020

Earlier work this paper cites.

A primer in BERTology: What we know about how BERT works

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020 · 2020

Earlier work this paper cites.

Which *bert? A survey organizing contextualized encoders

Patrick Xia, Shijie Wu, and Benjamin Van Durme. 2020 · 2020

Earlier work this paper cites.

NVIDIA A100 tensor core GPU: performance and innovation

Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021 · 2021

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021 · 2021

Earlier work this paper cites.

Does knowledge distillation really work?

Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson. 2021 · 2021

Earlier work this paper cites.

GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model

Ben Wang and Aran Komatsuzaki. 2021 · 2021

Earlier work this paper cites.

Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation

Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021 · 2021

Earlier work this paper cites.

Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression

Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, and Furu Wei. 2021 · 2021

Cited alongside, same era.

Gpt3.int8(): 8-bit matrix multiplication for transformers at scale

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022 · 2022

Cited alongside, same era.

Optimal brain compression: A framework for accurate post-training quantization and pruning

Elias Frantar and Dan Alistarh. 2022 · 2022

Cited alongside, same era.

In-context learning distillation: Transferring few-shot learning ability of pre-trained language models

Original

Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen R. McKeown. 2022 · 2022

Cited alongside, same era.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022

Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling

Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. 2023 · 2023

Closest in time.

How does calibration data affect the post-training pruning and quantization of large language models?

Original

Miles Williams and Nikolaos Aletras. 2023 · 2023

Closest in time.

Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity

Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. 2023 · 2023

Closest in time.

Smoothquant: Accurate and efficient post-training quantization for large language models

Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. 2023 · 2023

Closest in time.

Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation