Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

Representation degeneration problem in training natural language generation models

Original

Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019 · 1907

Earlier work this paper cites.

Roberta: A robustly optimized BERT pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a · 1907

Earlier work this paper cites.

Roberta: A robustly optimized BERT pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b · 1907

Earlier work this paper cites.

Codesearchnet challenge: Evaluating the state of semantic code search

Original

Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019 · 1909

Earlier work this paper cites.

Megatron-lm: Training multi-billion parameter language models using model parallelism

Original

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019 · 1909

Earlier work this paper cites.

On the difficulty of warm-starting neural network training

Original

Jordan T. Ash and Ryan P. Adams. 2019 · 1910

Earlier work this paper cites.

Glu variants improve transformer

Original

Noam Shazeer. 2020 · 2002

Earlier work this paper cites.

Longformer: The long-document transformer

Original

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020 · 2004

Earlier work this paper cites.

Improving low compute language modeling with in-domain embedding initialisation

Original

Charles Welch, Rada Mihalcea, and Jonathan K. Kummerfeld. 2020 · 2009

Earlier work this paper cites.

Ms marco: A human generated machine reading comprehension dataset

Original

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016 · 2016

Earlier work this paper cites.

Gaussian error linear units (gelus)

Original

Dan Hendrycks and Kevin Gimpel. 2016 · 2016

Earlier work this paper cites.

Layer normalization

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016 · 2016

Earlier work this paper cites.

Language modeling with gated convolutional networks

Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskeve. 2018 · 2018

Earlier work this paper cites.

Adafactor: Adaptive learning rates with sublinear memory cost

Noam Shazeer and Mitchell Stern. 2018 · 2018

Earlier work this paper cites.

Representation learning with contrastive predictive coding

Original

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018 · 2018

Earlier work this paper cites.

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018 · 2018

Earlier work this paper cites.

A broad-coverage challenge corpus for sentence understanding through inference

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018 · 2018

Earlier work this paper cites.

A walk with sgd

Original

Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. 2018 · 2018

Earlier work this paper cites.

BERT: pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Earlier work this paper cites.

Sentence-bert: Sentence embeddings using siamese bert-networks

Original

Nils Reimers and Iryna Gurevych. 2019 · 2019

Earlier work this paper cites.

Understanding and improving layer normalization

Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. 2019 · 2019

Earlier work this paper cites.

Language models are few-shot learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020

Earlier work this paper cites.

ELECTRA: pre-training text encoders as discriminators rather than generators

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020 · 2020

Earlier work this paper cites.

Dense passage retrieval for open-domain question answering

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2020

Earlier work this paper cites.

Colbert: Efficient and effective passage search via contextualized late interaction over BERT

Omar Khattab and Matei Zaharia. 2020 · 2020

Earlier work this paper cites.

Retrieval-augmented generation for knowledge-intensive nlp tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020 · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020

Earlier work this paper cites.

On layer normalization in the transformer architecture

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. 2020 · 2020

Earlier work this paper cites.

Hard negative examples are hard, but useful

Hong Xuan, Abby Stylianou, Xiaotong Liu, and Robert Pless. 2020 · 2020

Earlier work this paper cites.

Dall·e mini

Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phúc Lê Khăc, Luke Melas, and Ritobrata Ghosh. 2021 · 2021

Cited alongside, same era.

How to train BERT with an academic budget

Peter Izsak, Moshe Berchansky, and Omer Levy. 2021 · 2021

Cited alongside, same era.

composer

The Mosaic ML Team. 2021 · 2021

Cited alongside, same era.

Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth

Thao Nguyen, Maithra Raghu, and Simon Kornblith. 2021 · 2021

Cited alongside, same era.

BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021 · 2021

Cited alongside, same era.

Trec-covid: constructing a pandemic information retrieval test collection

Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021 · 2021

Should you mask 15% in masked language modeling?

Original

Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023 · 2023

Later among the works it cites.

Stable and low-precision training for large-scale vision-language models

Original

Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. 2023 · 2023

Later among the works it cites.

C-pack: Packaged resources to advance general chinese embedding

Original

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023 · 2023

Later among the works it cites.

React: Synergizing reasoning and acting in language models

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023 · 2023

Later among the works it cites.

A survey of large language models

Original