Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Boolq: Exploring the surprising difficulty of natural yes/no questions

Original

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019 · 1905

Earlier work this paper cites.

Hellaswag: Can a machine really finish your sentence?

Original

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 1905

Earlier work this paper cites.

Simple statistical gradient-following algorithms for connectionist reinforcement learning

Ronald J Williams. 1992 · 1992

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009

Earlier work this paper cites.

Parameter-exploring policy gradients

Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. 2010 · 2010

Earlier work this paper cites.

Analysis and improvement of policy gradient estimation

Tingting Zhao, Hirotaka Hachiya, Gang Niu, and Masashi Sugiyama. 2011 · 2011

Earlier work this paper cites.

Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application

Original

Weiran Wang and Miguel A Carreira-Perpinán. 2013 · 2013

Earlier work this paper cites.

Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding

Original

Song Han, Huizi Mao, and William J Dally. 2015 · 2015

Earlier work this paper cites.

The concrete distribution: A continuous relaxation of discrete random variables

Original

Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2016 · 2016

Earlier work this paper cites.

Pointer sentinel mixture models

Original

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016 · 2016

Earlier work this paper cites.

The lambada dataset: Word prediction requiring a broad discourse context

Original

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016 · 2016

Earlier work this paper cites.

Race: Large-scale reading comprehension dataset from examinations

Original

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017 · 2017

Earlier work this paper cites.

Training sparse neural networks

Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. 2017 · 2017

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Original

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 · 2018

Earlier work this paper cites.

The lottery ticket hypothesis: Finding sparse, trainable neural networks

Original

Jonathan Frankle and Michael Carbin. 2018 · 2018

Earlier work this paper cites.

Amc: Automl for model compression and acceleration on mobile devices

Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. 2018 · 2018

Earlier work this paper cites.

Data-driven sparse structure selection for deep neural networks

Zehao Huang and Naiyan Wang. 2018 · 2018

Earlier work this paper cites.

Metapruning: Meta learning for automatic neural network channel pruning

Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. 2019 · 2019

Earlier work this paper cites.

Importance estimation for neural network pruning

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019 · 2019

Earlier work this paper cites.

Piqa: Reasoning about physical commonsense in natural language

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020 · 2020

Earlier work this paper cites.

What is the state of neural network pruning?

Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. 2020 · 2020

Earlier work this paper cites.

Rigging the lottery: Making all tickets winners

Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020 · 2020

Earlier work this paper cites.

An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods

Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. 2020 · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020

Cited alongside, same era.

Winogrande: An adversarial winograd schema challenge at scale

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021 · 2021

Cited alongside, same era.

Neural pruning via growing regularization

Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. 2021 · 2021

Cited alongside, same era.

Extracting effective subnetworks with gumbel-softmax

Robin Dupont, Mohammed Amine Alaoui, Hichem Sahbi, and Alice Lebois. 2022 · 2022

Cited alongside, same era.

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022 · 2022

Cited alongside, same era.

A review of the gumbel-max trick and its extensions for discrete stochasticity in machine learning

Dynamic sparse no training: Training-free fine-tuning for sparse llms

Original

Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. 2023 · 2023

Later among the works it cites.

Fluctuation-based adaptive structured pruning for large language models

Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024 · 2024

Closest in time.

Slicegpt: Compress large language models by deleting rows and columns

Original

Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024 · 2024

Closest in time.

Sparsellm: Towards global pruning of pre-trained language models

Guangji Bai, Yijiang Li, Chen Ling, Kibaek Kim, and Liang Zhao. 2024 · 2024

Closest in time.

Fast and optimal weight update for pruned large language models

Original

Vladimír Boža. 2024 · 2024