ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018

Earlier work this paper cites.

Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks

Original

Jason Phang, Thibault Févry, and Samuel R Bowman · 2018

Earlier work this paper cites.

Glue: A multi-task benchmark and analysis platform for natural language understanding

Original

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman · 2018

Earlier work this paper cites.

Superglue: A stickier benchmark for general-purpose language understanding systems

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman · 2019

Earlier work this paper cites.

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020

Cited alongside, same era.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020

Cited alongside, same era.

Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He · 2020

Cited alongside, same era.

Transformers: State-of-the-art natural language processing

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush · 2020

Later among the works it cites.

How to train bert with an academic budget, 2021

Original

Peter Izsak, Moshe Berchansky, and Omer Levy · 2021

Later among the works it cites.

Eigencurve: Optimal learning rate schedule for SGD on quadratic objectives with skewed hessian spectrums

Original

Rui Pan, Haishan Ye, and Tong Zhang · 2021

Later among the works it cites.