2020

Hybrid Discriminative-Generative Training via Contrastive Learning

Liu, Hao, Abbeel, Pieter

Understand

Contrastive learning and supervised learning have both seen significant progress and success.

However, thus far they have largely been treated as two separate objectives, brought together only by having a shared neural network.
In this paper we show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning.
Beyond presenting this unified view, we show our specific choice of approximation of the energy-based loss outperforms the existing practice in terms of classification accuracy of WideResNet on CIFAR-10 and CIFAR-100.

Built on

A simple framework for contrastive learning of visual representations
Original
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G · 2002
Earlier work this paper cites.
Training products of experts by minimizing contrastive divergence
Hinton, G. E · 2002
Earlier work this paper cites.
On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes
Ng, A. Y. and Jordan, M. I · 2002
Earlier work this paper cites.
Improved baselines with momentum contrastive learning
Original
Chen, X., Fan, H., Girshick, R., and He, K · 2003
Earlier work this paper cites.
Classification with hybrid generative/discriminative models
Raina, R., Shen, Y., Mccallum, A., and Ng, A. Y · 2004
Earlier work this paper cites.
Big self-supervised models are strong semi-supervised learners
Original
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G · 2006
Earlier work this paper cites.
Principled hybrids of generative and discriminative models
Lasserre, J. A., Bishop, C. M., and Minka, T. P · 2006
Earlier work this paper cites.
A tutorial on energy-based learning
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F · 2006
Earlier work this paper cites.
Learning a nonlinear embedding by preserving class neighbourhood structure
Salakhutdinov, R. and Hinton, G · 2007
Earlier work this paper cites.
Learning generative models via discriminative approaches
Tu, Z · 2007
Earlier work this paper cites.
Classification using discriminative restricted boltzmann machines
Larochelle, H. and Bengio, Y · 2008
Earlier work this paper cites.
Training restricted boltzmann machines using approximations to the likelihood gradient
Tieleman, T · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. et al · 2009
Earlier work this paper cites.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. and Hyvärinen, A · 2010
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y · 2011
Earlier work this paper cites.
Bayesian learning via stochastic gradient langevin dynamics
Welling, M. and Teh, Y. W · 2011
Earlier work this paper cites.
Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics
Gutmann, M. U. and Hyvärinen, A · 2012
Earlier work this paper cites.

Similar

Distributed representations of words and phrases and their compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J · 2013
Cited alongside, same era.
Learning word embeddings efficiently with noise-contrastive estimation
Mnih, A. and Kavukcuoglu, K · 2013
Cited alongside, same era.
Explaining and harnessing adversarial examples
Original
Goodfellow, I. J., Shlens, J., and Szegedy, C · 2014
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K. and Zisserman, A · 2014
Cited alongside, same era.
Distilling the knowledge in a neural network
Original
Hinton, G., Vinyals, O., and Dean, J · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Ioffe, S. and Szegedy, C · 2015
Cited alongside, same era.
Deep learning face attributes in the wild
Liu, Z., Luo, P., Wang, X., and Tang, X · 2015
Cited alongside, same era.
Highway networks
Original
Srivastava, R. K., Greff, K., and Schmidhuber, J · 2015
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
A baseline for detecting misclassified and out-of-distribution examples in neural networks
Original
Hendrycks, D. and Gimpel, K · 2016
Cited alongside, same era.
Improved techniques for training gans
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X · 2016
Cited alongside, same era.
Rethinking the inception architecture for computer vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z · 2016
Cited alongside, same era.

Then

Towards deep learning models resistant to adversarial attacks
Original
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A · 2017
Later among the works it cites.
Glow: Generative flow with invertible 1x1 convolutions
Kingma, D. P. and Dhariwal, P · 2018
Later among the works it cites.
Spectral normalization for generative adversarial networks
Original
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y · 2018
Later among the works it cites.
Representation learning with contrastive predictive coding
Original
Oord, A. v. d., Li, Y., and Vinyals, O · 2018
Later among the works it cites.
Unsupervised feature learning via non-parametric instance-level discrimination
Original
Wu, Z., Xiong, Y., Yu, S., and Lin, D · 2018
Later among the works it cites.
Residual flows for invertible generative modeling
Chen, R. T., Behrmann, J., Duvenaud, D. K., and Jacobsen, J.-H · 2019
Later among the works it cites.
Implicit generation and modeling with energy based models
Du, Y. and Mordatch, I · 2019
Later among the works it cites.
Your classifier is secretly an energy based model and you should treat it like one
Original
Grathwohl, W., Wang, K.-C., Jacobsen, J.-H., Duvenaud, D., Norouzi, M., and Swersky, K · 2019
Later among the works it cites.
Video representation learning by dense predictive coding
Han, T., Xie, W., and Zisserman, A · 2019
Later among the works it cites.
Momentum contrast for unsupervised visual representation learning
Original
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R · 2019
Later among the works it cites.
When does label smoothing help?
Müller, R., Kornblith, S., and Hinton, G. E · 2019
Later among the works it cites.
Learning non-convergent non-persistent short-run mcmc toward energy-based model
Nijkamp, E., Hill, M., Zhu, S.-C., and Wu, Y. N · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al · 2019
Later among the works it cites.
Image synthesis with a single (robust) classifier
Santurkar, S., Ilyas, A., Tsipras, D., Engstrom, L., Tran, B., and Madry, A · 2019
Later among the works it cites.
Generative modeling by estimating gradients of the data distribution
Song, Y. and Ermon, S · 2019
Later among the works it cites.
Contrastive multiview coding
Original
Tian, Y., Krishnan, D., and Isola, P · 2019
Later among the works it cites.
Bootstrap your own latent: A new approach to self-supervised learning
Original
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al · 2020
Closest in time.
Supervised contrastive learning
Original
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…