2020

Debiased Contrastive Learning

Chuang, Ching-Yao, Robinson, Joshua, Yen-Chen, Lin et al.

Understand

A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples.

Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label.
Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available.
Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels.

Built on

A simple framework for contrastive learning of visual representations
Original
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton · 2002
Earlier work this paper cites.
Overview of trec 2002
Ellen M Voorhees · 2002
Earlier work this paper cites.
Improved baselines with momentum contrastive learning
Original
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He · 2003
Earlier work this paper cites.
Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources
Bill Dolan, Chris Quirk, and Chris Brockett · 2004
Earlier work this paper cites.
Mining and summarizing customer reviews
Minqing Hu and Bing Liu · 2004
Earlier work this paper cites.
A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts
Bo Pang and Lillian Lee · 2004
Earlier work this paper cites.
Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
Bo Pang and Lillian Lee · 2005
Earlier work this paper cites.
Annotating expressions of opinions and emotions in language
Janyce Wiebe, Theresa Wilson, and Claire Cardie · 2005
Earlier work this paper cites.
Dimensionality reduction by learning an invariant mapping
Raia Hadsell, Sumit Chopra, and Yann LeCun · 2006
Earlier work this paper cites.
Learning classifiers from only positive and unlabeled data
Charles Elkan and Keith Noto · 2008
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky et al · 2009
Earlier work this paper cites.
An analysis of single-layer networks in unsupervised feature learning
Adam Coates, Andrew Ng, and Honglak Lee · 2011
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean · 2013
Earlier work this paper cites.
Discriminative unsupervised feature learning with convolutional neural networks
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox · 2014
Earlier work this paper cites.
Analysis of learning from positive and unlabeled data
Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama · 2014
Earlier work this paper cites.

Similar

Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Cited alongside, same era.
Auto-encoding variational bayes
Diederik P Kingma and Max Welling · 2014
Cited alongside, same era.
Convex formulation for learning from positive and unlabeled data
Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama · 2015
Cited alongside, same era.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2015
Cited alongside, same era.
Skip-thought vectors
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler · 2015
Cited alongside, same era.
Infogan: Interpretable representation learning by information maximizing generative adversarial nets
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel · 2016
Cited alongside, same era.
Learning low-dimensional representations of medical concepts
Youngduck Choi, Chill Yi-I Chiu, and David Sontag · 2016
Cited alongside, same era.
Class-prior estimation for learning from positive and unlabeled data
Marthinus Christoffel, Gang Niu, and Masashi Sugiyama · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Estimating the class prior and posterior from noisy positives and unlabeled data
Shantanu Jain, Martha White, and Predrag Radivojac · 2016
Cited alongside, same era.
Unsupervised learning of visual representations by solving jigsaw puzzles
Mehdi Noroozi and Paolo Favaro · 2016
Cited alongside, same era.
Colorful image colorization
Richard Zhang, Phillip Isola, and Alexei A Efros · 2016
Cited alongside, same era.

Then

An efficient framework for learning sentence representations
Lajanugen Logeswaran and Honglak Lee · 2018
Later among the works it cites.
Deep learning for healthcare: review, opportunities and challenges
Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley · 2018
Later among the works it cites.
Representation learning with contrastive predictive coding
Original
Aaron van den Oord, Yazhe Li, and Oriol Vinyals · 2018
Later among the works it cites.
Deepmind control suite
Original
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al · 2018
Later among the works it cites.
A theoretical analysis of contrastive unsupervised representation learning
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi · 2019
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Later among the works it cites.
Data-efficient image recognition with contrastive predictive coding
Original
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord · 2019
Later among the works it cites.
A survey on bias and fairness in machine learning
Original
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan · 2019
Later among the works it cites.
Contrastive multiview coding
Original
Yonglong Tian, Dilip Krishnan, and Phillip Isola · 2019
Later among the works it cites.
S4l: Self-supervised semi-supervised learning
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer · 2019
Later among the works it cites.
Momentum contrast for unsupervised visual representation learning
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick · 2020
Closest in time.
Inorganic materials synthesis planning with literature-trained neural networks
Edward Kim, Zach Jensen, Alexander van Grootel, Kevin Huang, Matthew Staib, Sheshera Mysore, Haw-Shiuan Chang, Emma Strubell, Andrew McCallum, Stefanie Jegelka, et al · 2020
Closest in time.
Curl: Contrastive unsupervised representations for reinforcement learning
Original
Aravind Srinivas, Michael Laskin, and Pieter Abbeel · 2020
Closest in time.
A deep learning approach to antibiotic discovery
Jonathan M Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackerman, et al · 2020
Closest in time.
Understanding contrastive representation learning through alignment and uniformity on the hypersphere
Original
Tongzhou Wang and Phillip Isola · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…