2022

Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model

Xing, Yinghui, Wu, Qirui, Cheng, De et al.

Understand

With the emergence of large pre-trained vison-language model like CLIP, transferable representations can be adapted to a wide range of downstream tasks via prompt tuning.

Prompt tuning tries to probe the beneficial information for downstream tasks from the general knowledge stored in the pre-trained model.
A recently proposed method named Context Optimization (CoOp) introduces a set of learnable vectors as text prompt from the language side.
However, tuning the text prompt alone can only adjust the synthesized "classifier", while the computed visual features of the image encoder can not be affected , thus leading to sub-optimal solutions.

Built on

L. Fei-Fei, R. Fergus, and P. Perona, “Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,” in 2004 conference on computer vision and pattern recognition workshop . IEEE, 2004, pp. 178–178
2004
Earlier work this paper cites.
M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing . IEEE, 2008, pp. 722–729
2008
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255
2009
Earlier work this paper cites.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun database: Large-scale scene recognition from abbey to zoo,” in 2010 IEEE computer society conference on computer vision and pattern recognition . IEEE, 2010, pp. 3485–3492
2010
Earlier work this paper cites.
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, “Cats and dogs,” in 2012 IEEE conference on computer vision and pattern recognition . IEEE, 2012, pp. 3498–3505
2012
Earlier work this paper cites.
K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402 , 2012
Original
2012
Earlier work this paper cites.
S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine-grained visual classification of aircraft,” arXiv preprint arXiv:1306.5151 , 2013
Original
2013
Earlier work this paper cites.
J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in Proceedings of the IEEE international conference on computer vision workshops , 2013, pp. 554–561
2013
Earlier work this paper cites.
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 580–587
2014
Earlier work this paper cites.
L. Bossard, M. Guillaumin, and L. V. Gool, “Food-101–mining discriminative components with random forests,” in European conference on computer vision . Springer, 2014, pp. 446–461
2014
Earlier work this paper cites.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 3606–3613
2014
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
P. Jing, Y. Su, L. Nie, and H. Gu, “Predicting image memorability through adaptive transfer learning from external sources,” IEEE Transactions on Multimedia , vol. 19, no. 5, pp. 1050–1062, 2016
2016
Earlier work this paper cites.
S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Learning multiple visual domains with residual adapters,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , vol. 12, no. 7, pp. 2217–2226, 2019
2019
Earlier work this paper cites.
B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do imagenet classifiers generalize to imagenet?” in International Conference on Machine Learning . PMLR, 2019, pp. 5389–5400
2019
Earlier work this paper cites.
H. Wang, S. Ge, Z. Lipton, and E. P. Xing, “Learning robust global representations by penalizing local predictive power,” Advances in Neural Information Processing Systems , vol. 32, 2019
2019
Earlier work this paper cites.
Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” arXiv preprint arXiv:2010.00747 , 2020
Original
2020
Earlier work this paper cites.
T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh, “Autoprompt: Eliciting knowledge from language models with automatically generated prompts,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020, pp. 4222–4235
2020
Earlier work this paper cites.
Z. Jiang, F. F. Xu, J. Araki, and G. Neubig, “How can we know what language models know?” Transactions of the Association for Computational Linguistics , vol. 8, pp. 423–438, 2020
2020
Earlier work this paper cites.
Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in European conference on computer vision . Springer, 2020, pp. 104–120
2020
Earlier work this paper cites.

Similar

Y. Lu, W. Wang, C. Yuan, X. Li, and Z. Lai, “Manifold transfer learning via discriminant regression analysis,” IEEE Transactions on Multimedia , vol. 23, pp. 2056–2070, 2020
2020
Cited alongside, same era.
J. O. Zhang, A. Sax, A. Zamir, L. Guibas, and J. Malik, “Side-tuning: a baseline for network adaptation via additive side networks,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16 . Springer, 2020, pp. 698–714
2020
Cited alongside, same era.
H. Cai, C. Gan, L. Zhu, and S. Han, “Tinytl: Reduce memory, not parameters for efficient on-device learning,” Advances in Neural Information Processing Systems , vol. 33, pp. 11 285–11 297, 2020
2020
Cited alongside, same era.
Y. Zhu, W. Min, and S. Jiang, “Attribute-guided feature learning for few-shot image recognition,” IEEE Transactions on Multimedia , vol. 23, pp. 1200–1209, 2020
2020
Cited alongside, same era.
H. Huang, J. Zhang, J. Zhang, J. Xu, and Q. Wu, “Low-rank pairwise alignment bilinear network for few-shot fine-grained image classification,” IEEE Transactions on Multimedia , vol. 23, pp. 1666–1680, 2020
2020
Cited alongside, same era.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 9729–9738
2020
Cited alongside, same era.
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al. , “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems , vol. 33, pp. 21 271–21 284, 2020
2020
Cited alongside, same era.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning . PMLR, 2021, pp. 8748–8763
2021
Cited alongside, same era.
C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in International Conference on Machine Learning . PMLR, 2021, pp. 4904–4916
2021
Cited alongside, same era.
K. Desai and J. Johnson, “Virtex: Learning visual representations from textual annotations,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 11 162–11 173
2021
Cited alongside, same era.
B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” arXiv preprint arXiv:2104.08691 , 2021
Original
2021
Cited alongside, same era.
Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, “Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm,” arXiv preprint arXiv:2110.05208 , 2021
Original
2021
Cited alongside, same era.

Then

H. Fang, P. Xiong, L. Xu, and W. Luo, “Transferring image-clip to video-text retrieval via temporal relations,” IEEE Transactions on Multimedia , 2022
2022
Closest in time.
J. Yang, Z. Li, F. Zheng, A. Leonardis, and J. Song, “Prompting for multi-modal tracking,” in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 3492–3500
2022
Closest in time.
Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu, “Denseclip: Language-guided dense prediction with context-aware prompting,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 18 082–18 091
2022
Closest in time.
R. Zhang, Z. Zeng, Z. Guo, and Y. Li, “Can language understand depth?” in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 6868–6874
2022
Closest in time.
R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li, “Pointclip: Point cloud understanding by clip,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 8552–8562
2022
Closest in time.
M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” arXiv preprint arXiv:2203.12119 , 2022
Original
2022
Closest in time.
Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister, “Learning to prompt for continual learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 139–149
2022
Closest in time.
H. Bahng, A. Jahanian, S. Sankaranarayanan, and P. Isola, “Exploring visual prompts for adapting large-scale models,” arXiv preprint arXiv:2203.17274 , 2022
Original
2022
Closest in time.
B. Zhu, Y. Niu, Y. Han, Y. Wu, and H. Zhang, “Prompt-aligned gradient for prompt tuning,” arXiv preprint arXiv:2205.14865 , 2022
Original
2022
Closest in time.
Y. Lu, J. Liu, Y. Zhang, Y. Liu, and X. Tian, “Prompt distribution learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5206–5215
2022
Closest in time.
H. Bahng, A. Jahanian, S. Sankaranarayanan, and P. Isola, “Exploring visual prompts for adapting large-scale models,” arXiv preprint arXiv:2203.17274 , vol. 1, no. 3, p. 4, 2022
Original
2022
Closest in time.
Y. Zhang, K. Zhou, and Z. Liu, “Neural prompt search,” arXiv preprint arXiv:2206.04673 , 2022
Original
2022
Closest in time.
B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, and H. Ling, “Expanding language-image pretrained models for general video recognition,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IV . Springer, 2022, pp. 1–18
2022
Closest in time.
D. Li, J. Li, H. Li, J. C. Niebles, and S. C. Hoi, “Align and prompt: Video-and-language pre-training with entity prompts,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 4953–4963
2022
Closest in time.
H. Yang, J. Lin, A. Yang, P. Wang, C. Zhou, and H. Yang, “Prompt tuning for generative multimodal pretrained models,” arXiv preprint arXiv:2208.02532 , 2022
Original
2022
Closest in time.
H. Zhang, H. Li, and P. Koniusz, “Multi-level second-order few-shot learning,” IEEE Transactions on Multimedia , 2022
2022
Closest in time.
K. Guo, C. Shen, B. Hu, M. Hu, and X. Kui, “Rsnet: relation separation network for few-shot similar class recognition,” IEEE Transactions on Multimedia , 2022
2022
Closest in time.
P. Tian, H. Yu, and S. Xie, “An adversarial meta-training framework for cross-domain few-shot learning,” IEEE Transactions on Multimedia , 2022
2022
Closest in time.
X. Zhong, C. Gu, M. Ye, W. Huang, and C.-W. Lin, “Graph complemented latent representation for few-shot image classification,” IEEE Transactions on Multimedia , 2022
2022
Closest in time.
L. Zhang, Y. Du, J. Shen, and X. Zhen, “Learning to learn with variational inference for cross-domain image classification,” IEEE Transactions on Multimedia , 2022
2022
Closest in time.
K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 000–16 009
2022
Closest in time.
H. Cheng, J. T. Zhou, W. P. Tay, and B. Wen, “Graph neural networks with triple attention for few-shot learning,” IEEE Transactions on Multimedia , 2023
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…