2022

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Li, Junnan, Li, Dongxu, Xiong, Caiming et al.

Understand

Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks.

However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks.
Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision.
In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks.

Built on

Im2text: Describing images using 1 million captioned photographs
Ordonez, V., Kulkarni, G., and Berg, T. L · 2011
Earlier work this paper cites.
Microsoft COCO: common objects in context
Lin, T., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L · 2014
Earlier work this paper cites.
VQA: visual question answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D · 2015
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Hinton, G., Vinyals, O., and Dean, J · 2015
Earlier work this paper cites.
Deep visual-semantic alignments for generating image descriptions
Karpathy, A. and Li, F · 2015
Earlier work this paper cites.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S · 2015
Earlier work this paper cites.
Visual dialog
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., and Batra, D · 2017
Earlier work this paper cites.
Making the V in VQA matter: Elevating the role of image understanding in visual question answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D · 2017
Earlier work this paper cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L., Shamma, D. A., Bernstein, M. S., and Fei-Fei, L · 2017
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Loshchilov, I. and Hutter, F · 2017
Earlier work this paper cites.
Bilinear attention networks
Kim, J., Jun, J., and Zhang, B · 2018
Earlier work this paper cites.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning
Sharma, P., Ding, N., Goodman, S., and Soricut, R · 2018
Earlier work this paper cites.
nocaps: novel object captioning at scale
Agrawal, H., Anderson, P., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., and Lee, S · 2019
Earlier work this paper cites.
BERT: pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M., Lee, K., and Toutanova, K · 2019
Earlier work this paper cites.
Heterogeneous memory enhanced multimodal attention model for video question answering
Fan, C., Zhang, X., Zhang, S., Wang, W., Zhang, C., and Huang, H · 2019
Earlier work this paper cites.
Multi-step reasoning via recurrent dual attention for visual dialog
Gan, Z., Cheng, Y., Kholy, A. E., Li, L., Liu, J., and Gao, J · 2019
Earlier work this paper cites.
Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks
Lu, J., Batra, D., Parikh, D., and Lee, S · 2019
Earlier work this paper cites.
Pytorch: An imperative style, high-performance deep learning library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al · 2019
Earlier work this paper cites.

Similar

A survey on image data augmentation for deep learning
Shorten, C. and Khoshgoftaar, T. M · 2019
Cited alongside, same era.
A corpus for reasoning about natural language grounded in photographs
Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., and Artzi, Y · 2019
Cited alongside, same era.
Do not have enough data? deep learning to the rescue!
Anaby-Tavor, A., Carmeli, B., Goldbraich, E., Kantor, A., Kour, G., Shlomov, S., Tepper, N., and Zwerdling, N · 2020
Cited alongside, same era.
UNITER: universal image-text representation learning
Chen, Y., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J · 2020
Cited alongside, same era.
e-snli-ve: Corrected visual-textual entailment with natural language explanations
Original
Do, V., Camburu, O.-M., Akata, Z., and Lukasiewicz, T · 2020
Cited alongside, same era.
Large-scale adversarial training for vision-and-language representation learning
Gan, Z., Chen, Y., Li, L., Zhu, C., Cheng, Y., and Liu, J · 2020
Cited alongside, same era.
The curious case of neural text degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y · 2020
Cited alongside, same era.
Data augmentation using pre-trained transformer models
Original
Kumar, V., Choudhary, A., and Cho, E · 2020
Cited alongside, same era.
Hierarchical conditional relation networks for video question answering
Le, T. M., Le, V., Venkatesh, S., and Tran, T · 2020
Cited alongside, same era.
Oscar: Object-semantics aligned pre-training for vision-language tasks
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J · 2020
Cited alongside, same era.
End-to-end learning of visual representations from uncurated instructional videos
Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Sivic, J., and Zisserman, A · 2020
Cited alongside, same era.
Large-scale pretraining for visual dialog: A simple state-of-the-art baseline
Murahari, V., Batra, D., Parikh, D., and Das, A · 2020
Cited alongside, same era.

Then

Actbert: Learning global-local video-text representations
Zhu, L. and Yang, Y · 2020
Later among the works it cites.
Frozen in time: A joint video and image encoder for end-to-end retrieval
Bain, M., Nagrani, A., Varol, G., and Zisserman, A · 2021
Later among the works it cites.
Is space-time attention all you need for video understanding?
Bertasius, G., Wang, H., and Torresani, L · 2021
Later among the works it cites.
Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R · 2021
Later among the works it cites.
Unifying vision-and-language tasks via text generation
Original
Cho, J., Lei, J., Tan, H., and Bansal, M · 2021
Later among the works it cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N · 2021
Later among the works it cites.
Scaling up vision-language pre-training for image captioning, 2021
Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., and Wang, L · 2021
Later among the works it cites.
Scaling up visual and vision-language representation learning with noisy text supervision
Original
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T · 2021
Later among the works it cites.
Vilt: Vision-and-language transformer without convolution or region supervision
Original
Kim, W., Son, B., and Kim, I · 2021
Later among the works it cites.
Less is more: Clipbert for video-and-language learning via sparse sampling
Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T. L., Bansal, M., and Liu, J · 2021
Later among the works it cites.
Support-set bottlenecks for video-text representation learning
Patrick, M., Huang, P.-Y., Asano, Y., Metze, F., Hauptmann, A. G., Henriques, J. F., and Vedaldi, A · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Original
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al · 2021
Later among the works it cites.
Laion-400m: Open dataset of clip-filtered 400 million image-text pairs
Original
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A · 2021
Later among the works it cites.
Simvlm: Simple visual language model pretraining with weak supervision
Original
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y · 2021
Later among the works it cites.
Videoclip: Contrastive pre-training for zero-shot video-text understanding
Xu, H., Ghosh, G., Huang, P.-Y., Okhonko, D., Aghajanyan, A., Metze, F., Zettlemoyer, L., and Feichtenhofer, C · 2021
Later among the works it cites.
Just ask: Learning to answer questions from millions of narrated videos
Yang, A., Miech, A., Sivic, J., Laptev, I., and Schmid, C · 2021
Later among the works it cites.
Vinvl: Making visual representations matter in vision-language models
Original
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., and Gao, J · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…