Fetching the paper…
Reading the bibliography…
We introduce EfficientCL, a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning.
Language models are few-shot learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
A simple framework for contrastive learning of visual representations
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b · 2002
Earlier work this paper cites.
Big self-supervised models are strong semi-supervised learners
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. 2020c · 2006
Earlier work this paper cites.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020 · 2009
Earlier work this paper cites.
Coda: Contrast-enhanced and diversity-promoting data augmentation for natural language understanding
Yanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev, Jiawei Han, and Weizhu Chen. 2020 · 2010
Earlier work this paper cites.
Contrastive learning with hard negative samples
Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2020 · 2010
Earlier work this paper cites.
Supervised contrastive learning for pre-trained language model fine-tuning
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2020 · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012 · 2012
Cited alongside, same era.
Clear: Contrastive learning for sentence representation
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020 · 2012
Cited alongside, same era.
Curriculm adversarial training
Qi-Zhi Cai, Chang Liu, and Dawn Song. 2018 · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanov. 2019 · 2019
Cited alongside, same era.
On the power of curriculum learning in training deep networks
Guy Hacohen and Daphna Weinshall. 2019 · 2019
Cited alongside, same era.
Hard negative mixing for contrastive learning
Declutr: Deep contrastive learning for unsupervised textual representations
John M Giorgi, Osvald Nitski, Gary D. Bader, and Bo Wang. 2021 · 2021
Closest in time.
Curriculum learning: A regularization method for efficient and stable billion-scale gpt model pre-training
Conglong Li, Minjia Zhang, and Yuxiong He. 2021 · 2021
Closest in time.
Coco-lm: Correcting and contrasting text sequences for language model pretraining
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021 · 2021
Closest in time.
Shortformer: Better language modeling using shorter inputs
Ofir Press, Noah A. Smith, and Mike Lewis. 2021 · 2021
Closest in time.
Few-shot text classification with triplet networks, data augmentation, and curriculum learning
Jason Wei, Chengyu Huang, Soroush Vosoughi, Yu Cheng, and Shiqi Xu. 2021 · 2021
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. 2020 · 2020
Cited alongside, same era.
Curriculum learning for natural language understanding
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020 · 2020
Cited alongside, same era.
MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020a
Cited in the paper.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021 · 2021
Closest in time.
An embarrassingly simple approach for transfer learning from pretrained language models
A. Chronopoulou, C. Baziotis, and A. Potamianos. 2019 · 2095
Closest in time.