Fetching the paper…
Reading the bibliography…
Modern natural language processing (NLP) methods employ self-supervised pretraining objectives such as masked language modeling to boost the performance of various application tasks.
Loss functions for discriminative training of energy-based models
Yann LeCun and Fu Jie Huang · 2005
Earlier work this paper cites.
A tutorial on energy-based learning
Yann Lecun, Sumit Chopra, Raia Hadsell, Marc Aurelio Ranzato, and Fu Jie Huang · 2006
Earlier work this paper cites.
A fast and simple algorithm for training neural probabilistic language models
Andriy Mnih and Yee Whye Teh · 2012
Earlier work this paper cites.
Efficient estimation of word representations in vector space
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean · 2013
Earlier work this paper cites.
An efficient framework for learning sentence representations
Lajanugen Logeswaran and Honglak Lee · 2018
Earlier work this paper cites.
Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency
Zhuang Ma and Michael Collins · 2018
Earlier work this paper cites.
Representation learning with contrastive predictive coding
Aäron van den Oord, Yazhe Li, and Oriol Vinyals · 2018
Earlier work this paper cites.
Contrastive attention mechanism for abstractive sentence summarization
Xiangyu Duan, Hongfei Yu, Mingming Yin, Min Zhang, Weihua Luo, and Yue Zhang · 2019
Earlier work this paper cites.
Learning deep representations by mutual information estimation and maximization
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio · 2019
Earlier work this paper cites.
Transferable contrastive network for generalized zero-shot learning
Huajie Jiang, Ruiping Wang, Shiguang Shan, and Xilin Chen · 2019
Earlier work this paper cites.
Roberta: A robustly optimized BERT pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Earlier work this paper cites.
GILE: A Generalized Input-Label Embedding for Text Classification
Nikolaos Pappas and James Henderson · 2019
Earlier work this paper cites.
The MuCoW test suite at WMT 2019: Automatically harvested multilingual contrastive word sense disambiguation test sets for machine translation
Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann · 2019
Earlier work this paper cites.
A theoretical analysis of contrastive unsupervised representation learning
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar · 2019
Cited alongside, same era.
On Losses for Modern Language Models
Stéphane Aroca-Ouellette and Frank Rudzicz · 2020
Cited alongside, same era.
Are all negatives created equal in contrastive instance discrimination?, 2020
Tiffany Tianhui Cai, Jonathan Frankle, David J. Schwab, and Ari S. Morcos · 2020
Cited alongside, same era.
MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification
Jiaao Chen, Zichao Yang, and Diyi Yang · 2020
Cited alongside, same era.
Big self-supervised models are strong semi-supervised learners
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton · 2020
Cited alongside, same era.
Pre-training transformers as energy-based cloze models
A metric learning reality check
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim · 2020
Later among the works it cites.
Long-tail zero and few-shot learning via contrastive pretraining on and for small data, 2020
Nils Rethmeier and Isabelle Augenstein · 2020
Later among the works it cites.
Contrastive distillation on intermediate representations for language model compression
Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu · 2020
Later among the works it cites.
Learning with contrastive examples for data-to-text generation
Yui Uehara, Tatsuya Ishigaki, Kasumi Aoki, Hiroshi Noji, Keiichi Goshima, Ichiro Kobayashi, Hiroya Takamura, and Yusuke Miyao · 2020
Later among the works it cites.
Clear: Contrastive learning for sentence representation, 2020
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma · 2020
Later among the works it cites.
A survey on contrastive self-supervised learning
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Kevin Clark, Minh-Thang Luong, Quoc Le, and Christopher D. Manning · 2020
Cited alongside, same era.
Residual energy-based models for text generation
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato · 2020
Cited alongside, same era.
CERT: contrastive self-supervised learning for language understanding
Hongchao Fang and Pengtao Xie · 2020
Cited alongside, same era.
Declutr: Deep contrastive learning for unsupervised textual representations, 2020
John M. Giorgi, Osvald Nitski, Gary D. Bader, and Bo Wang · 2020
Cited alongside, same era.
Bootstrap your own latent - a new approach to self-supervised learning
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko · 2020
Cited alongside, same era.
Pretraining with contrastive sentence objectives improves discourse performance of language models
Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky · 2020
Cited alongside, same era.
Supervised contrastive learning
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan · 2020
Cited alongside, same era.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon · 2021
Closest in time.
Scaling up visual and vision-language representation learning with noisy text supervision, 2021
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig · 2021
Closest in time.
COCO-LM: correcting and contrasting text sequences for language model pretraining
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song · 2021
Closest in time.
On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow · 2021
Closest in time.
CoDA: Contrast-enhanced and diversity-promoting data augmentation for natural language understanding
Yanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev, Weizhu Chen, and Jiawei Han · 2021
Closest in time.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever · 2021
Closest in time.
Contrasting distinct structured views to learn sentence embeddings, 2021
Antoine Simoulin and Benoit Crabbé · 2021
Closest in time.
Contrastive learning inverts the data generating process, 2021
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel · 2021
Closest in time.