Landscape-Aware Growing: The Power of a Little LAG

The cascade-correlation learning architecture

Scott Fahlman and Christian Lebiere · 1989

Earlier work this paper cites.

Knowledge transfer in deep convolutional neural nets

Steven Gutstein, Olac Fuentes, and Eric Freudenthal · 2008

Earlier work this paper cites.

Net2net: Accelerating learning via knowledge transfer

Original

Tianqi Chen, Ian Goodfellow, and Jonathon Shlens · 2015

Earlier work this paper cites.

Net2net: Accelerating learning via knowledge transfer

Tianqi Chen, Ian Goodfellow, and Jonathon Shlens · 2016

Earlier work this paper cites.

Network morphism

Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen · 2016

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018

Earlier work this paper cites.

Adafactor: Adaptive learning rates with sublinear memory cost

Noam Shazeer and Mitchell Stern · 2018

Earlier work this paper cites.

Masked structural growth for 2x faster language model pre-training

Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang · 2018

Cited alongside, same era.

Efficient training of bert by progressively stacking

Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu · 2019

Cited alongside, same era.

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter · 2019

Cited alongside, same era.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020

Cited alongside, same era.

bert2BERT: Towards reusable pretrained language models

Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu · 2022

Cited alongside, same era.

Gradmax: Growing neural networks using gradient information

Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Fabian Pedregosa, and Max Vladymyrov · 2022

Staged training for transformer language models

Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy · 2022

Later among the works it cites.

Ul2: Unifying language learning paradigms

Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al · 2022

Later among the works it cites.

Learning to grow pretrained models for efficient transformer training

Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim · 2022

Later among the works it cites.

Flm-101b: An open llm and how to train it with $100 k budget

Original

Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, et al · 2023

Later among the works it cites.

Efficient training of language models using few-shot learning

Sashank Reddi, Sobhan Miryoosefi, Stefani Karp, Shankar Krishnan, Satyen Kale, Seungyeon Kim, and Sanjiv Kumar · 2023

Later among the works it cites.