2020

VirTex: Learning Visual Representations from Textual Annotations

Desai, Karan, Johnson, Justin

Understand

The de-facto approach to many vision tasks is to start from pretrained visual representations, typically learned via supervised training on ImageNet.

Recent methods have explored unsupervised pretraining to scale to vast quantities of unlabeled images.
In contrast, we aim to learn high-quality visual representations from fewer images.
To this end, we revisit supervised pretraining, and seek data-efficient alternatives to classification-based pretraining.

Built on

Boris T Polyak, “Some methods of speeding up the convergence of iteration methods,”
1964
Earlier work this paper cites.
Raia Hadsell, Sumit Chopra, and Yann LeCun, “Dimensionality reduction by learning an invariant mapping,” in
2006
Earlier work this paper cites.
Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin, “LIBLINEAR: A library for large linear classification,”
2008
Earlier work this paper cites.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in
2009
Earlier work this paper cites.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman, “The pascal visual object classes (VOC) challenge,”
2009
Earlier work this paper cites.
Michael Gutmann and Aapo Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in
2010
Earlier work this paper cites.
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg, “Im2Text: Describing images using 1 million captioned photographs,” in
2011
Earlier work this paper cites.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,”
2011
Earlier work this paper cites.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, “Imagenet classification with deep convolutional neural networks,” in
2012
Earlier work this paper cites.
James Martens and Ilya Sutskever, “Training deep and recurrent networks with hessian-free optimization,” in
2012
Earlier work this paper cites.
Micah Hodosh, Peter Young, and Julia Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,”
2013
Earlier work this paper cites.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton, “On the importance of initialization and momentum in deep learning,” in
2013
Earlier work this paper cites.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell, “Decaf: A deep convolutional activation feature for generic visual recognition,” in
2014
Earlier work this paper cites.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson, “Cnn features off-the-shelf: an astounding baseline for recognition,” in
2014
Earlier work this paper cites.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in
2014
Earlier work this paper cites.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick, “Microsoft COCO: Common objects in context,” in
2014
Earlier work this paper cites.
Jia Deng, Olga Russakovsky, Jonathan Krause, Michael S Bernstein, Alex Berg, and Li Fei-Fei, “Scalable multi-label annotation,” in
2014
Earlier work this paper cites.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,”
2014
Earlier work this paper cites.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg, “Referitgame: Referring to objects in photographs of natural scenes,” in
2014
Earlier work this paper cites.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,”
2014
Earlier work this paper cites.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein,
2015
Earlier work this paper cites.
Jonathan Long, Evan Shelhamer, and Trevor Darrell, “Fully convolutional networks for semantic segmentation,” in
2015
Earlier work this paper cites.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan, “Show and tell: A neural image caption generator,” in
2015
Earlier work this paper cites.
Andrej Karpathy and Li Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in
2015
Earlier work this paper cites.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in
2015
Earlier work this paper cites.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh, “VQA: Visual question answering,” in
2015
Earlier work this paper cites.
Carl Doersch, Abhinav Gupta, and Alexei A Efros, “Unsupervised visual representation learning by context prediction,” in
2015
Earlier work this paper cites.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick, “Microsoft COCO captions: Data collection and evaluation server,”
Original
2015
Earlier work this paper cites.
Sergey Ioffe and Christian Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in
2015
Earlier work this paper cites.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh, “CIDEr: Consensus-based image description evaluation,” in
2015
Earlier work this paper cites.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in
2015
Earlier work this paper cites.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in
2016
Earlier work this paper cites.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei, “Visual7w: Grounded question answering in images,” in
2016
Earlier work this paper cites.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros, “Context encoders: Feature learning by inpainting,” in
2016
Earlier work this paper cites.
Richard Zhang, Phillip Isola, and Alexei A Efros, “Colorful image colorization,” in
2016
Earlier work this paper cites.
Ranjay A Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A Shamma, Li Fei-Fei, and Michael S Bernstein, “Embracing error to enable rapid crowdsourcing,” in
2016
Earlier work this paper cites.
Junhua Mao, Jiajing Xu, Kevin Jing, and Alan L Yuille, “Training and evaluating multimodal word embeddings with large-scale web annotated images,” in
2016
Earlier work this paper cites.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li, “YFCC100M: The new data in multimedia research,”
2016
Earlier work this paper cites.
Mehdi Noroozi and Paolo Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in
2016
Earlier work this paper cites.

Similar

Dan Hendrycks and Kevin Gimpel, “Gaussian error linear units (gelus),”
Original
2016
Cited alongside, same era.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton, “Layer normalization,”
Original
2016
Cited alongside, same era.
Hakan Inan, Khashayar Khosravi, and Richard Socher, “Tying word vectors and word classifiers: A loss framework for language modeling,”
Original
2016
Cited alongside, same era.
Rico Sennrich, Barry Haddow, and Alexandra Birch, “Neural machine translation of rare words with subword units,” in
2016
Cited alongside, same era.
Ilya Loshchilov and Frank Hutter, “SGDR: Stochastic gradient descent with warm restarts,”
Original
2016
Cited alongside, same era.
Sergey Zagoruyko and Nikos Komodakis, “Wide residual networks,”
Original
2016
Cited alongside, same era.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, “SPICE: Semantic propositional image caption evaluation,” in
2016
Cited alongside, same era.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin, “Training deep nets with sublinear memory cost,”
Original
2016
Cited alongside, same era.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, “Mask R-CNN,” in
2017
Cited alongside, same era.
Richard Zhang, Phillip Isola, and Alexei A Efros, “Split-brain autoencoders: Unsupervised learning by cross-channel prediction,” in
2017
Cited alongside, same era.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” in
2017
Cited alongside, same era.
Ang Li, Allan Jabri, Armand Joulin, and Laurens van der Maaten, “Learning visual n-grams from web data,” in
2017
Cited alongside, same era.

Then

Jeff Donahue and Karen Simonyan, “Large scale adversarial representation learning,” in
2019
Later among the works it cites.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang, “Unsupervised embedding learning via invariant and spreading instance feature,” in
2019
Later among the works it cites.
Philip Bachman, R Devon Hjelm, and William Buchwalter, “Learning representations by maximizing mutual information across views,” in
2019
Later among the works it cites.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio, “Learning deep representations by mutual information estimation and maximization,”
2019
Later among the works it cites.
Trieu H Trinh, Minh-Thang Luong, and Quoc V Le, “Selfie: Self-supervised pretraining for image embedding,”
Original
2019
Later among the works it cites.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins, “Local aggregation for unsupervised learning of visual embeddings,” in
2019
Later among the works it cites.
Drew A Hudson and Christopher D Manning, “GQA: A new dataset for real-world visual reasoning and compositional question answering,” in
2019
Later among the works it cites.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi, “A corpus for reasoning about natural language grounded in photographs,” in
2019
Later among the works it cites.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi, “From recognition to cognition: Visual commonsense reasoning,” in
2019
Later among the works it cites.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in
2019
Later among the works it cites.
Hao Tan and Mohit Bansal, “LXMERT: Learning cross-modality encoder representations from transformers,” in
2019
Later among the works it cites.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee, “ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” in
2019
Later among the works it cites.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang, “VisualBERT: A simple and performant baseline for vision and language,”
Original
2019
Later among the works it cites.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu, “Uniter: Learning universal image-text representations,”
Original
2019
Later among the works it cites.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le, “XLNet: Generalized autoregressive pretraining for language understanding,” in
2019
Later among the works it cites.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, “Language models are unsupervised multitask learners,”
2019
Later among the works it cites.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov, “RoBERTa: A robustly optimized bert pretraining approach,”
Original
2019
Later among the works it cites.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro, “Megatron-LM: Training multi-billion parameter language models using model parallelism,”
Original
2019
Later among the works it cites.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton, “Lookahead optimizer: k steps forward, 1 step back,” in
2019
Later among the works it cites.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in
2019
Later among the works it cites.
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson, “nocaps: novel object captioning at scale,” in
2019
Later among the works it cites.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick, “Detectron2.”
2019
Later among the works it cites.
Kaiming He, Ross Girshick, and Piotr Dollár, “Rethinking ImageNet pre-training,” in
2019
Later among the works it cites.
Yonglong Tian, Dilip Krishnan, and Phillip Isola, “Contrastive multiview coding,” in
2020
Closest in time.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, “Momentum contrast for unsupervised visual representation learning,” in
2020
Closest in time.
Ishan Misra and Laurens van der Maaten, “Self-supervised learning of pretext-invariant representations,” in
2020
Closest in time.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton, “A simple framework for contrastive learning of visual representations,” in
2020
Closest in time.
Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoc V Le, “Self-training with noisy student improves ImageNet classification,” in
2020
Closest in time.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C.H. Hoi, “Prototypical contrastive learning of unsupervised representations,”
Original
2020
Closest in time.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” in
2020
Closest in time.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai, “VL-BERT: Pre-training of generic visual-linguistic representations,” 2020
2020
Closest in time.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou, “Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training,”
2020
Closest in time.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao, “Unified vision-language pre-training for image captioning and VQA,”
2020
Closest in time.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei,
2020
Closest in time.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus, “Learning visual representations with caption annotations,” in
2020
Closest in time.
Jonathan C Stroud, David A Ross, Chen Sun, Jia Deng, Rahul Sukthankar, and Cordelia Schmid, “Learning video representations from textual web supervision,” in
2020
Closest in time.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Original
2020
Closest in time.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning, “ELECTRA: Pre-training text encoders as discriminators rather than generators,” in
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…