Pre-training Distillation for Large Language Models: A Design Space Exploration

A study of bfloat16 for deep learning training

Original

Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. 2019 · 1905

Earlier work this paper cites.

Adaptive regularization of labels

Original

Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Shu-Tao Xia. 2019 · 1908

Earlier work this paper cites.

Adam: A method for stochastic optimization

Original

Diederik P Kingma. 2014 · 2014

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

Geoffrey Hinton. 2015 · 2015

Earlier work this paper cites.

Distilling knowledge from ensembles of neural networks for speech recognition

Yevgen Chebotar and Austin Waters. 2016 · 2016

Earlier work this paper cites.

Overview of the nlpcc-iccpol 2016 shared task: Open domain chinese question answering

Nan Duan. 2016 · 2016

Earlier work this paper cites.

Sequence-level knowledge distillation

Yoon Kim and Alexander M Rush. 2016 · 2016

Earlier work this paper cites.

Overview of the nlpcc 2017 shared task: Open domain chinese question answering

Nan Duan and Duyu Tang. 2018 · 2017

Earlier work this paper cites.

Efficient knowledge distillation from an ensemble of teachers

Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramabhadran. 2017 · 2017

Earlier work this paper cites.

Transferring knowledge to smaller network with class-distance loss

Seungwook Kim and Hyo-Eun Kim. 2017 · 2017

Earlier work this paper cites.

Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer

Nikos Komodakis and Sergey Zagoruyko. 2017 · 2017

Earlier work this paper cites.

A gift from knowledge distillation: Fast optimization, network minimization and transfer learning

Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. 2017 · 2017

Earlier work this paper cites.

Variational information distillation for knowledge transfer

Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. 2019 · 2019

Earlier work this paper cites.

The curious case of neural text degeneration

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019 · 2019

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

When does label smoothing help?

Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019 · 2019

Earlier work this paper cites.

Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019 · 2019

Earlier work this paper cites.

Energy and policy considerations for deep learning in nlp

Emma Strubell, Ananya Ganesh, and Andrew Mccallum. 2019 · 2019

Earlier work this paper cites.

Hellaswag: Can a machine really finish your sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 2019

Earlier work this paper cites.

Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings

Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. 2020 · 2020

Earlier work this paper cites.

Piqa: Reasoning about physical commonsense in natural language

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020 · 2020

Earlier work this paper cites.

Distilling knowledge learned in bert for text generation

Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2020 · 2020

Earlier work this paper cites.

Dynabert: Dynamic bert with adaptive width and depth

Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020 · 2020

Earlier work this paper cites.

Tinybert: Distilling bert for natural language understanding

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020 · 2020

Cited alongside, same era.

Fastbert: a self-distilling bert with adaptive inference time

Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020 · 2020

Cited alongside, same era.

Improved knowledge distillation via teacher assistant

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020 · 2020

Cited alongside, same era.

Winogrande: An adversarial winograd schema challenge at scale

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020 · 2020

Cited alongside, same era.

Contrastive representation distillation

Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020 · 2020

Cited alongside, same era.

Exclusivity-consistency regularized knowledge distillation for face recognition

Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality

Vicuna. 2023 · 2023

Later among the works it cites.

Wizardlm: Empowering large language models to follow complex instructions

Original

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023 · 2023

Later among the works it cites.

Weak-to-strong generalization: Eliciting strong capabilities with weak supervision

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. 2024 · 2024

Closest in time.

Ultrafeedback: Boosting language models with scaled ai feedback

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. 2024 · 2024

Closest in time.

Language modeling is compression

Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. 2024 · 2024

Closest in time.