Fetching the paper…
Reading the bibliography…
How can neural networks trained by contrastive learning extract features from the unlabeled data? Why does contrastive learning usually need much stronger data augmentations than supervised learning to ensure good representations? These questions involve both the optimization and statistical aspects of deep learning, but can hardly be answered by analyzing supervised learning, where the target functions are the highest pursuit.
Sparse coding with an overcomplete basis set: A strategy employed by v1 ?
Bruno A. Olshausen and David J. Field · 1997
Earlier work this paper cites.
Sparse coding in the primate cortex
Peter Földiák and Malcolm P. Young · 1998
Earlier work this paper cites.
Sparse coding and decorrelation in primary visual cortex during natural vision
William E. Vinje and Jack L. Gallant · 2000
Earlier work this paper cites.
Backward feature correction: How deep learning performs deep learning
Zeyuan Allen-Zhu and Yuanzhi Li · 2001
Earlier work this paper cites.
Sparse coding of sensory inputs
Bruno A Olshausen and David J Field · 2004
Earlier work this paper cites.
Feature purification: How adversarial training performs robust deep learning
Zeyuan Allen-Zhu and Yuanzhi Li · 2005
Earlier work this paper cites.
On contrastive divergence learning
Miguel Á. Carreira-Perpiñán and Geoffrey E. Hinton · 2005
Earlier work this paper cites.
Contrastive estimation: Training log-linear models on unlabeled data
Noah A. Smith and Jason Eisner · 2005
Earlier work this paper cites.
Image sequence denoising via sparse and redundant representations
M. Protter and M. Elad · 2009
Earlier work this paper cites.
Linear spatial pyramid matching using sparse coding for image classification
Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang · 2009
Earlier work this paper cites.
Understanding self-supervised learning with dual deep networks
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli · 2010
Earlier work this paper cites.
An analysis of single-layer networks in unsupervised feature learning
Adam Coates, Andrew Y. Ng, and Honglak Lee · 2011
Earlier work this paper cites.
Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics
Michael U Gutmann and Aapo Hyvärinen · 2012
Earlier work this paper cites.
Efficient estimation of word representations in vector space
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean · 2013
Earlier work this paper cites.
Sparse Modeling for Image and Vision Processing
Julien Mairal, Francis Bach, and Jean Ponce · 2014
Earlier work this paper cites.
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le · 2014
Earlier work this paper cites.
Globally optimal gradient descent for a convnet with gaussian inputs
Alon Brutzkus and Amir Globerson · 2017
Earlier work this paper cites.
Convergence analysis of two-layer neural networks with relu activation
Yuanzhi Li and Yang Yuan · 2017
Earlier work this paper cites.
Learning relus via gradient descent
Mahdi Soltanolkotabi · 2017
Cited alongside, same era.
Linear algebraic structure of word senses, with applications to polysemy
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski · 2018
Cited alongside, same era.
Learning one-hidden-layer neural networks with landscape design
Rong Ge, Jason D. Lee, and Tengyu Ma · 2018
Cited alongside, same era.
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Cited alongside, same era.
Learning overparameterized neural networks via stochastic gradient descent on structured data
Yuanzhi Li and Yingyu Liang · 2018
Cited alongside, same era.
Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang · 2018
Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
Yu Bai and Jason D. Lee · 2020
Later among the works it cites.
A simple framework for contrastive learning of visual representations
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton · 2020
Later among the works it cites.
Big Self-Supervised Models are Strong Semi-Supervised Learners
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton · 2020
Later among the works it cites.
Exploring Simple Siamese Representation Learning
Xinlei Chen and Kaiming He · 2020
Later among the works it cites.
Bootstrap Your Own Latent A New Approach to Self-Supervised Learning
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko · 2020
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Representation learning with contrastive predictive coding
Aaron van den Oord, Yazhe Li, and Oriol Vinyals · 2018
Cited alongside, same era.
Memorization in overparameterized autoencoders
Adityanarayanan Radhakrishnan, Karren Yang, Mikhail Belkin, and Caroline Uhler · 2018
Cited alongside, same era.
What can resnet learn efficiently, going beyond kernels?
Zeyuan Allen-Zhu and Yuanzhi Li · 2019
Cited alongside, same era.
Learning and generalization in overparameterized neural networks, going beyond two layers
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang · 2019
Cited alongside, same era.
A convergence theory for deep learning via over-parameterization
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song · 2019
Cited alongside, same era.
A theoretical analysis of contrastive unsupervised representation learning
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi · 2019
Cited alongside, same era.
Later among the works it cites.
Momentum contrast for unsupervised visual representation learning
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick · 2020
Later among the works it cites.
Self-supervised visual feature learning with deep neural networks: A survey
Longlong Jing and Yingli Tian · 2020
Later among the works it cites.
Adversarial self-supervised contrastive learning
Minseon Kim, Jihoon Tack, and Sung Ju Hwang · 2020
Later among the works it cites.
Predicting What You Already Know Helps: Provable Self-Supervised Learning
Jason D. Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo · 2020
Later among the works it cites.
Learning over-parametrized two-layer relu neural networks beyond ntk
Yuanzhi Li, Tengyu Ma, and Hongyang R. Zhang · 2020
Later among the works it cites.
Contrastive estimation reveals topic posterior information to linear models
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu · 2020
Later among the works it cites.
Demystifying self-supervised learning: An information-theoretical framework
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency · 2020
Later among the works it cites.
Understanding contrastive representation learning through alignment and uniformity on the hypersphere
Tongzhou Wang and Phillip Isola · 2020
Later among the works it cites.
Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma · 2020
Later among the works it cites.
Zeyuan Allen-Zhu and Yuanzhi Li · 2021
Closest in time.
Provable guarantees for self-supervised deep learning with spectral contrastive loss
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma · 2021
Closest in time.
Contrastive learning, multi-view redundancy, and linear models
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu · 2021
Closest in time.