2021

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Kim, Wonjae, Son, Bokyung, Kim, Ildoo

Understand

Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks.

Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet).
Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary.
In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs.

Built on

Improved baselines with momentum contrastive learning
Original
Chen, X., Fan, H., Girshick, R., and He, K · 2003
Earlier work this paper cites.
Im2text: Describing images using 1 million captioned photographs
Ordonez, V., Kulkarni, G., and Berg, T · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Microsoft coco: Common objects in context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L · 2014
Earlier work this paper cites.
Deep visual-semantic alignments for generating image descriptions
Karpathy, A. and Fei-Fei, L · 2015
Earlier work this paper cites.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Faster r-cnn: Towards real-time object detection with region proposal networks
Ren, S., He, K., Girshick, R., and Sun, J · 2016
Earlier work this paper cites.
Vse++: Improving visual-semantic embeddings with hard negatives
Original
Faghri, F., Fleet, D. J., Kiros, J. R., and Fidler, S · 2017
Earlier work this paper cites.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D · 2017
Earlier work this paper cites.
Mask r-cnn
He, K., Gkioxari, G., Dollár, P., and Girshick, R · 2017
Earlier work this paper cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Earlier work this paper cites.
Aggregated residual transformations for deep neural networks
Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K · 2017
Earlier work this paper cites.
mixup: Beyond empirical risk minimization
Original
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D · 2017
Earlier work this paper cites.
Bottom-up and top-down attention for image captioning and visual question answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L · 2018
Earlier work this paper cites.
Deep clustering for unsupervised learning of visual features
Caron, M., Bojanowski, P., Joulin, A., and Douze, M · 2018
Earlier work this paper cites.
Pythia v0. 1: the winning entry to the vqa challenge 2018
Original
Jiang, Y., Natarajan, V., Chen, X., Rohrbach, M., Batra, D., and Parikh, D · 2018
Earlier work this paper cites.

Similar

Stacked cross attention for image-text matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., and He, X · 2018
Cited alongside, same era.
Decoupled weight decay regularization
Loshchilov, I. and Hutter, F · 2018
Cited alongside, same era.
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A · 2018
Cited alongside, same era.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning
Sharma, P., Ding, N., Goodman, S., and Soricut, R · 2018
Cited alongside, same era.
A corpus for reasoning about natural language grounded in photographs
Original
Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., and Artzi, Y · 2018
Cited alongside, same era.
Self-labelling via simultaneous clustering and representation learning
Asano, Y., Rupprecht, C., and Vedaldi, A · 2019
Cited alongside, same era.
Multigrain: a unified image embedding for classes and instances
Original
Berman, M., Jégou, H., Vedaldi, A., Kokkinos, I., and Douze, M · 2019
Cited alongside, same era.
Unsupervised pre-training of image features on non-curated data
Caron, M., Bojanowski, P., Mairal, J., and Joulin, A · 2019
Cited alongside, same era.
Uniter: Learning universal image-text representations
Original
Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J · 2019
Cited alongside, same era.
Pre-training with whole word masking for chinese bert
Original
Cui, Y., Che, W., Liu, T., Qin, B., Yang, Z., Wang, S., and Hu, G · 2019
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2019
Cited alongside, same era.
Visualbert: A simple and performant baseline for vision and language
Original
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W · 2019
Cited alongside, same era.

Then

Unsupervised learning of visual features by contrasting cluster assignments
Original
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A · 2020
Later among the works it cites.
X-lxmert: Paint, caption and answer questions with multi-modal transformers
Cho, J., Lu, J., Schwenk, D., Hajishirzi, H., and Kembhavi, A · 2020
Later among the works it cites.
Randaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V · 2020
Later among the works it cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al · 2020
Later among the works it cites.
Large-scale adversarial training for vision-and-language representation learning
Original
Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., and Liu, J · 2020
Later among the works it cites.
Momentum contrast for unsupervised visual representation learning
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R · 2020
Later among the works it cites.
Augment your batch: Improving generalization through instance repetition
Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., and Soudry, D · 2020
Later among the works it cites.
Pixel-bert: Aligning image pixels with text by deep multi-modal transformers
Original
Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J · 2020
Later among the works it cites.
In defense of grid features for visual question answering
Jiang, H., Misra, I., Rohrbach, M., Learned-Miller, E., and Chen, X · 2020
Later among the works it cites.
The open images dataset v4
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al · 2020
Later among the works it cites.
12-in-1: Multi-task vision and language representation learning
Lu, J., Goswami, V., Rohrbach, M., Parikh, D., and Lee, S · 2020
Later among the works it cites.
Revisiting modulated convolutions for visual counting and beyond
Original
Nguyen, D.-K., Goswami, V., and Chen, X · 2020
Later among the works it cites.
Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data
Original
Qi, D., Su, L., Song, J., Cui, E., Bharti, T., and Sacheti, A · 2020
Later among the works it cites.
Training data-efficient image transformers & distillation through attention
Original
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H · 2020
Later among the works it cites.
A fast proximal point method for computing exact wasserstein distance
Xie, Y., Wang, X., Wang, R., and Zha, H · 2020
Later among the works it cites.
Ernie-vil: Knowledge enhanced vision-language representations through scene graph
Original
Yu, F., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H., and Wang, H · 2020
Later among the works it cites.
Unified vision-language pre-training for image captioning and vqa
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J. J., and Gao, J · 2020
Later among the works it cites.
Learning transferable visual models from natural language supervision, 2021
Radford, A., Sutskever, I., Kim, J., Krueger, G., and Agarwal, S · 2021
Closest in time.
Vinvl: Making visual representations matter in vision-language models
Original
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., and Gao, J · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…