2021

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP

Fürst, Andreas, Rumetshofer, Elisabeth, Lehner, Johannes et al.

Understand

CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3.

CLIP vision models that have a rich representation are pre-trained using the InfoNCE objective and natural language supervision before they are fine-tuned on particular tasks.
Though CLIP excels at zero-shot transfer learning, it suffers from an explaining away problem, that is, it focuses on one or few features, while neglecting other relevant features.
This problem is caused by insufficiently extracting the covariance structure in the original multi-modal data.

Built on

A simple test for uniformity of a circular distribution
Ajne, B · 1968
Earlier work this paper cites.
Learning patterns and pattern sequences by self-organizing nets of threshold elements
Amari, S.-I · 1972
Earlier work this paper cites.
On invariant tests of uniformity for directions and orientations
Prentice, M. J · 1978
Earlier work this paper cites.
Neural networks and physical systems with emergent collective computational abilities
Hopfield, J. J · 1982
Earlier work this paper cites.
Neurons with graded response have collective computational properties like those of two-state neurons
Hopfield, J. J · 1984
Earlier work this paper cites.
High order correlation model for associative memory
Chen, H. H., Lee, Y. C., Sun, G. Z., Lee, H. Y., Maxwell, T., and Giles, C. L · 1986
Earlier work this paper cites.
Nonlinear discriminant functions and associative memories
Psaltis, D. and Cheol, H. P · 1986
Earlier work this paper cites.
Storage capacity of generalized networks
Abbott, L. F. and Arian, Y · 1987
Earlier work this paper cites.
Number of stable points for spin-glasses and neural networks of higher orders
Baldi, P. and Venkatesh, S. S · 1987
Earlier work this paper cites.
Multiconnected neural network models
Gardner, E · 1987
Earlier work this paper cites.
Capacities of multiconnected memory models
Horn, D. and Usher, M · 1988
Earlier work this paper cites.
Embracing causality in default reasoning
Pearl, J · 1988
Earlier work this paper cites.
Explaining ’explaining away’
Wellman, M. P. and Henrion, M · 1993
Earlier work this paper cites.
Storage capacity of kernel associative memories
Caputo, B. and Niemann, H · 2002
Earlier work this paper cites.
Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories
Fei-Fei, L., Fergus, R., and Perona, P · 2004
Earlier work this paper cites.
Automated flower classification over a large number of classes
Nilsback, M.-E. and Zisserman, A · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A · 2009
Earlier work this paper cites.
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., and Harmeling, S · 2009
Earlier work this paper cites.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. and Hyvärinen, A · 2010
Earlier work this paper cites.
Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization
Nguyen, X., Wainwright, M. J., and Jordan, M · 2010
Earlier work this paper cites.
NIST handbook of mathematical functions
Olver, F. W. J., Lozier, D. W., Boisvert, R. F., and Clark, C. W · 2010
Earlier work this paper cites.
Sun database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A · 2010
Earlier work this paper cites.
An Analysis of Single Layer Networks in Unsupervised Feature Learning
Coates, A., Ng, A., and Lee, H · 2011
Earlier work this paper cites.
The German traffic sign recognition benchmark: A multi-class classification competition
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C · 2011
Earlier work this paper cites.
Cats and dogs
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V · 2012
Earlier work this paper cites.
Conceptual short term memory in perception and thought
Potter, M · 2012
Earlier work this paper cites.
A dataset of 101 human action classes from videos in the wild
Soomro, K., Zamir, A. R., and Shah, M · 2012
Earlier work this paper cites.
Challenges in representation learning: A report on three machine learning contests, 2013
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J., Ionescu, R., Popescu, M., Grozea, C., Bergstra, J., Xie, J., Romaszko, L., Xu, B., Chuang, Z., and Bengio, Y · 2013
Earlier work this paper cites.
3D object representations for fine-grained categorization
Krause, J., Stark, M., Deng, J., and Fei-Fei, L · 2013
Earlier work this paper cites.
Fine-grained visual classification of aircraft
Maji, S., Kannala, J., Rahtu, E., Blaschko, M., and Vedaldi, A · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J · 2013
Earlier work this paper cites.
Birdsnap: Large-scale fine-grained visual categorization of birds
Berg, T., Liu, J., Lee, S. W., Alexander, M. L., Jacobs, D. W., and Belhumeur, P. N · 2014
Earlier work this paper cites.
Food-101 – mining discriminative components with random forests
Bossard, L., Guillaumin, M., and Van Gool, L · 2014
Earlier work this paper cites.
Describing textures in the wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , and Vedaldi, A · 2014
Earlier work this paper cites.
Semi-supervised learning with deep generative models
Kingma, D. P., Mohamed, S., Rezende, D. J., and Welling, M · 2014
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Dense associative memory for pattern recognition
Krotov, D. and Hopfield, J. J · 2016
Earlier work this paper cites.

Similar

YFCC100M: The new data in multimedia research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J · 2016
Cited alongside, same era.
On sampling strategies for neural network-based collaborative filtering
Chen, T., Sun, Y., Shi, Y., and Hong, L · 2017
Cited alongside, same era.
Remote sensing image scene classification: Benchmark and state of the art
Cheng, G., Han, J., and Lu, X · 2017
Cited alongside, same era.
On the properties of the softmax function with application in game theory and reinforcement learning
Gao, B. and Pavel, L · 2017
Cited alongside, same era.
Efficient natural language response suggestion for smart reply
Henderson, M. L., Al-Rfou, R., Strope, B., Sung, Y.-H., Lukács, L., Guo, R., Kumar, S., Miklos, B., and Kurzweil, R · 2017
Cited alongside, same era.
SGDR: Stochastic gradient descent with warm restarts
Loshchilov, I. and Hutter, F · 2017
Cited alongside, same era.
Automatic differentiation in pytorch
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A · 2017
Cited alongside, same era.
Mutual information neural estimation
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R. D · 2018
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.
Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification
Helber, P., Bischke, B., Dengel, A., and Borth, D · 2018
Cited alongside, same era.
An efficient framework for learning sentence representations
Logeswaran, L. and Lee, H · 2018
Cited alongside, same era.
Formal limitations on the measurement of mutual information
McAllester, D. and Stratos, K · 2018
Cited alongside, same era.

Then

Understanding contrastive representation learning through alignment and uniformity on the hypersphere
Wang, T. and Isola, P · 2020
Later among the works it cites.
Modern Hopfield networks and attention for immune repertoire classification
Widrich, M., Schäfl, B., Pavlović, M., Ramsauer, H., Gruber, L., Holzleitner, M., Brandstetter, J., Sandve, G. K., Greiff, V., Hochreiter, S., and Klambauer, G · 2020
Later among the works it cites.
Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy
Yang, K., Qinami, K., Fei-Fei, L., Deng, J., and Russakovsky, O · 2020
Later among the works it cites.
Evaluating CLIP: Towards characterization of broader capabilities and downstream implications
Agarwal, S., Krueger, G., Clark, J., Radford, A., Kim, J. W., and Brundage, M · 2021
Closest in time.
Paint by word
Bau, D., Andonian, A., Cui, A., Park, Y., Jahanian, A., Oliva, A., and Torralba, A · 2021
Closest in time.
On the opportunities and risks of foundation models
Bommasani, R. et al · 2021
Closest in time.
Object representations in the human brain reflect the co-occurrence statistics of vision and language
Bonner, M. F. and Epstein, R. A · 2021
Closest in time.
Poisoning and backdooring contrastive learning
Carlini, N. and Terzis, A · 2021
Closest in time.
Simpler, faster, stronger: Breaking the log-K curse on contrastive learners with FlatNCE
Chen, J., Gan, Z., Li, X., Guo, Q., Chen, L., Gao, S., Chung, T., Xu, Y., Zeng, B., Lu, W., Li, F., Carin, L., and Tao, C · 2021
Closest in time.
Exploring simple siamese representation learning
Chen, X. and He, K · 2021
Closest in time.
Does language help generalization in vision models?
Devillers, B., Bielawski, R., Choski, B., and VanRullen, R · 2021
Closest in time.
CLIP2Video: Mastering video-text retrieval via image CLIP
Fang, H., Xiong, P., Xu, L., and Chen, Y · 2021
Closest in time.
CLIPDraw: Exploring text-to-drawing synthesis through language-image encoders
Frans, K., Soros, L. B., and Witkowski, O · 2021
Closest in time.
Generating images from caption and vice versa via CLIP-guided generative latent space search
Galatolo, F. A., Cimino, M. G. C. A., and Vaglini, G · 2021
Closest in time.
SimCSE: Simple contrastive learning of sentence embeddings
Gao, T., Yao, X., and Chen, D · 2021
Closest in time.
OpenCLIP, 2021
Ilharco, G., Wortsman, M., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L · 2021
Closest in time.
Prototypical contrastive learning of unsupervised representations
Li, J., Zhou, P., Xiong, C., Socher, R., and Hoi, S. C. H · 2021
Closest in time.
CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval
Luo, H., Ji, L., Zhong, M., Chen, Y., Lei, W., Duan, N., and Li, T · 2021
Closest in time.
Characterizing generalization under out-of-distribution shifts in deep metric learning
Milbich, T., Roth, K., Sinha, S., Schmidt, L., Ghassemi, M., and Ommer, B · 2021
Closest in time.
Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generalization
Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L · 2021
Closest in time.
CLIP-It! Language-guided video summarization
Narasimhan, M., Rohrbach, A., and Darrell, T · 2021
Closest in time.
Segmentation in style: Unsupervised semantic image segmentation with stylegan and CLIP
Pakhomov, D., Hira, S., Wagle, N., Green, K. E., and Navab, N · 2021
Closest in time.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I · 2021
Closest in time.
Hopfield networks is all you need
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S · 2021
Closest in time.
Modern Hopfield networks for few- and zero-shot reaction prediction
Seidl, P., Renz, P., Dyubankova, N., Neves, P., Verhoeven, J., Wegner, J. K., Hochreiter, S., and Klambauer, G · 2021
Closest in time.
How much can CLIP benefit vision-and-language tasks?
Shen, S., Li, L. H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K · 2021
Closest in time.
Conditional contrastive learning: Removing undesirable information in self-supervised representations
Tsai, Y.-H. H., Ma, M. Q., Zhao, H., Zhang, K., Morency, L.-P., and Salakhutdinov, R · 2021
Closest in time.
Understanding the behaviour of contrastive loss
Wang, F. and Liu, H · 2021
Closest in time.
Modern hopfield networks for return decomposition for delayed rewards
Widrich, M., Hofmarcher, M., Patil, V. P., Bitto-Nemling, A., and Hochreiter, S · 2021
Closest in time.
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Li, M., Kim, J. W., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L · 2021
Closest in time.
Conditional negative sampling for contrastive learning of visual representations
Wu, M., Mosse, M., Zhuang, C., Yamins, D., and Goodman, N · 2021
Closest in time.
Decoupled contrastive learning
Yeh, C.-H., Hong, C.-Y., Hsu, Y.-C., Liu, T.-L., Chen, Y., and LeCun, Y · 2021
Closest in time.
Learning to prompt for vision-language models
Zhou, K., Yang, J., Loy, C. C., and Liu, Z · 2021
Closest in time.
Understanding dimensional collapse in contrastive self-supervised learning
Jing, L., Vincent, P., LeCun, Y., and Tian, Y · 2022
Closest in time.
History compression via language models in reinforcement learning
Paischer, F., Adler, T., Patil, V., Bitto-Nemling, A., Holzleitner, M., Lehner, S., Eghbal-Zadeh, H., and Hochreiter, S · 2022
Closest in time.
Improving few-and zero-shot reaction template prediction using modern Hopfield networks
Seidl, P., Renz, P., Dyubankova, N., Neves, P., Verhoeven, J., Wegner, J. K., Segler, M., Hochreiter, S., and Klambauer, G · 2022
Closest in time.
Dual temperature helps contrastive learning without many negative samples: Towards understanding and simplifying moco
Zhang, C., Zhang, K., Pham, T. X., Niu, A., Qiao, Z., Yoo, C. D., and Kweon, I. S · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…