Built on

Y. LeCun, J. Denker, and S. Solla, “Optimal brain damage,” Advances in neural information processing systems
1989
Earlier work this paper cites.
M. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a large annotated corpus of english: The penn treebank,” 1993
1993
Earlier work this paper cites.
J. H. Friedman, “Greedy function approximation: a gradient boosting machine,” Annals of statistics
2001
Earlier work this paper cites.
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” in Proceedings of the IEEE international conference on computer vision
2015
Earlier work this paper cites.
S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843
Original
2016
Earlier work this paper cites.
R. Luo, F. Tian, T. Qin, E. Chen, and T.-Y. Liu, “Neural architecture optimization,” Advances in neural information processing systems
2018
Earlier work this paper cites.
P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz, “Importance estimation for neural network pruning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
2019
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al
2020
Earlier work this paper cites.
R. Luo, X. Tan, R. Wang, T. Qin, E. Chen, and T.-Y. Liu, “Accuracy prediction with non-neural model for neural architecture search,” arXiv preprint arXiv:2007.04785
Original
2020
Earlier work this paper cites.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300
Original
2020
Earlier work this paper cites.
Y. Hu, X. Wang, L. Li, and Q. Gu, “Improving one-shot nas with shrinking-and-expanding supernet,” Pattern Recognition
2021
Earlier work this paper cites.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al
2022
Earlier work this paper cites.

Similar

L. Li and Z. Jin, “Shadow knowledge distillation: Bridging offline and online knowledge transfer,” Advances in Neural Information Processing Systems
2022
Cited alongside, same era.
L. Li, L. Shiuan-Ni, Y. Yang, and Z. Jin, “Boosting online feature transfer via separable feature fusion.,” in IJCNN
2022
Cited alongside, same era.
L. Li, “Self-regulated feature learning via teacher-free feature distillation,” in ECCV 2022
2022
Cited alongside, same era.
L. Li, L. Shiuan-Ni, Y. Yang, and Z. Jin, “Teacher-free distillation via regularizing intermediate representation,” in IJCNN
2022
Cited alongside, same era.
E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323
Original
2022
Cited alongside, same era.

Then

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

Pruning Large Language Models via Accuracy Predictor

Ji, Yupeng, Cao, Yibo, Liu, Jiucai

Understand

Large language models(LLMs) containing tens of billions of parameters (or even more) have demonstrated impressive capabilities in various NLP tasks.

However, substantial model size poses challenges to training, inference, and deployment so that it is necessary to compress the model.
At present, most model compression for LLMs requires manual design of pruning features, which has problems such as complex optimization pipeline and difficulty in retaining the capabilities of certain parts of the model.Therefore, we propose a novel pruning approach: firstly, a training set of a certain number of architecture-accuracy pairs is established, and then a non-neural model is trained as an accuracy predictor.
Using the accuracy predictor to further optimize the search space and search, the optimal model can be automatically selected.

Built on

Y. LeCun, J. Denker, and S. Solla, “Optimal brain damage,” Advances in neural information processing systems
1989
Earlier work this paper cites.
M. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a large annotated corpus of english: The penn treebank,” 1993
1993
Earlier work this paper cites.
J. H. Friedman, “Greedy function approximation: a gradient boosting machine,” Annals of statistics
2001
Earlier work this paper cites.
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” in Proceedings of the IEEE international conference on computer vision
2015
Earlier work this paper cites.
S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843
Original
2016
Earlier work this paper cites.
R. Luo, F. Tian, T. Qin, E. Chen, and T.-Y. Liu, “Neural architecture optimization,” Advances in neural information processing systems
2018
Earlier work this paper cites.
P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz, “Importance estimation for neural network pruning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
2019
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al
2020
Earlier work this paper cites.
R. Luo, X. Tan, R. Wang, T. Qin, E. Chen, and T.-Y. Liu, “Accuracy prediction with non-neural model for neural architecture search,” arXiv preprint arXiv:2007.04785
Original
2020
Earlier work this paper cites.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300
Original
2020
Earlier work this paper cites.
Y. Hu, X. Wang, L. Li, and Q. Gu, “Improving one-shot nas with shrinking-and-expanding supernet,” Pattern Recognition
2021
Earlier work this paper cites.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al
2022
Earlier work this paper cites.

Similar

L. Li and Z. Jin, “Shadow knowledge distillation: Bridging offline and online knowledge transfer,” Advances in Neural Information Processing Systems
2022
Cited alongside, same era.
L. Li, L. Shiuan-Ni, Y. Yang, and Z. Jin, “Boosting online feature transfer via separable feature fusion.,” in IJCNN
2022
Cited alongside, same era.
L. Li, “Self-regulated feature learning via teacher-free feature distillation,” in ECCV 2022
2022
Cited alongside, same era.
L. Li, L. Shiuan-Ni, Y. Yang, and Z. Jin, “Teacher-free distillation via regularizing intermediate representation,” in IJCNN
2022
Cited alongside, same era.
E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323
Original
2022
Cited alongside, same era.

Then

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

P. Dong, X. Niu, L. Li, L. Xie, W. Zou, T. Ye, Z. Wei, and H. Pan, “Prior-guided one-shot neural architecture search,” arXiv preprint arXiv:2206.13329

Original

2022

Cited alongside, same era.

K. Chen, L. Yang, Y. Chen, K. Chen, Y. Xu, and L. Li, “Gp-nas-ensemble: a model for the nas performance prediction,” in CVPRW

2022

Cited alongside, same era.

Z. Wei, H. Pan, L. L. Li, M. Lu, X. Niu, P. Dong, and D. Li, “Convformer: Closing the gap between cnn and vision transformers,” arXiv preprint arXiv:2209.07738

Original

2022

Cited alongside, same era.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al

Original

2023

Cited alongside, same era.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al

Original

2023

Cited alongside, same era.

L. Li, P. Dong, Z. Wei, and Y. Yang, “Automated knowledge distillation via monte carlo tree search,” in ICCV

2023

Cited alongside, same era.

Original

2023

Closest in time.

M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A simple and effective pruning approach for large language models,” arXiv preprint arXiv:2306.11695

Original

2023

Closest in time.

E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” 2023

2023

Closest in time.

P. Dong, L. Li, and Z. Wei, “Diswot: Student architecture search for distillation without training,” in CVPR

2023

Closest in time.

P. Dong, X. Niu, L. Li, Z. Tian, X. Wang, Z. Wei, H. Pan, and D. Li, “Rd-nas: Enhancing one-shot supernet ranking ability via ranking distillation from zero-cost proxies,” in ICASSP)

2023

Closest in time.

H. Zhao, H. Chen, F. Yang, N. Liu, H. Deng, H. Cai, S. Wang, D. Yin, and M. Du, “Explainability for large language models: A survey,” arXiv preprint arXiv:2309.01029

Original

2023

Closest in time.

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” arXiv preprint arXiv:2305.14314

Original

2023

Closest in time.