Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments

Language models are few-shot learners

Tom Brown and 1 others. 2020 · 1901

Earlier work this paper cites.

Optimal brain damage

Yann LeCun, John S Denker, and Sara A Solla. 1990 · 1990

Earlier work this paper cites.

Optimal brain surgeon: Extensions and performance comparisons

Babak Hassibi, David G Stork, and Gregory J Wolff. 1993 · 1993

Earlier work this paper cites.

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997

Earlier work this paper cites.

Learning phrase representations using rnn encoder-decoder for statistical machine translation

Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014

Earlier work this paper cites.

Learning both weights and connections for efficient neural networks

Song Han, Jeff Pool, John Tran, and William J Dally. 2015 · 2015

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015 · 2015

Earlier work this paper cites.

Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding

Song Han, Huizi Mao, and William J Dally. 2016 · 2016

Earlier work this paper cites.

Pruning filters for efficient convnets

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016 · 2016

Earlier work this paper cites.

Pointer sentinel mixture models

Original

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016 · 2016

Earlier work this paper cites.

On large-batch training for deep learning: Generalization gap and sharp minima

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2017 · 2017

Earlier work this paper cites.

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

To prune, or not to prune: Exploring the efficacy of pruning for model compression

Original

Michael H Zhu and Suyog Gupta. 2017 · 2017

Earlier work this paper cites.

Deep rewiring: Training very sparse deep networks

Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. 2018 · 2018

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Earlier work this paper cites.

The lottery ticket hypothesis: Finding sparse, trainable neural networks

Jonathan Frankle and Michael Carbin. 2018 · 2018

Earlier work this paper cites.

Amc: Automl for model compression and acceleration on mobile devices

Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. 2018 · 2018

Earlier work this paper cites.

Quantization and training of neural networks for efficient integer-arithmetic-only inference

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018 · 2018

Cited alongside, same era.

Snip: Single-shot network pruning based on connection sensitivity

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. 2018 · 2018

Cited alongside, same era.

Benchmarking neural network robustness to common corruptions and perturbations

Dan Hendrycks and Thomas Dietterich. 2019 · 2019

Cited alongside, same era.

Are sixteen heads really better than one?

Paul Michel, Omer Levy, and Graham Neubig. 2019 · 2019

Cited alongside, same era.

Importance estimation for neural network pruning

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Irene Frosio, and Jan Kautz. 2019 · 2019

Cited alongside, same era.

Energy and policy considerations for deep learning in nlp

Practical issues in recurrent neural network pruning

Shaobo Liu, Ang Li, Bowen Feng, Feng Zhang, and Zewen Zhang. 2021 · 2021

Later among the works it cites.

Informer: Beyond efficient transformer for long sequence time-series forecasting

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021 · 2021

Later among the works it cites.

S4nd: Modeling images and videos as multidimensional signals using state spaces

Original

Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. 2022 · 2022

Later among the works it cites.

Diagonal state spaces are as effective as structured state spaces

Original

Ankit Gupta, Aditya Kusupati, Haricharan Simhadri, Harsh Pathak, Aniruddha Kembhavi, and Jitendra Malik. 2022 · 2022

Later among the works it cites.

Liquid structural state-space models

Original

Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Max Tschernuth, Joshua Tenenbaum, and Daniela Rus. 2022 · 2022