Visual Prompting in Multimodal Large Language Models: A Survey

Fast r-cnn

Ross Girshick. 2015 · 2015

Earlier work this paper cites.

Segformer: Simple and efficient design for semantic segmentation with transformers

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. 2021 · 2021

Earlier work this paper cites.

Visual prompting via image inpainting

Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Earlier work this paper cites.

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022 · 2022

Earlier work this paper cites.

Self-instruct: Aligning language models with self-generated instructions

Original

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022 · 2022

Earlier work this paper cites.

Omni3d: A large benchmark and model for 3d object detection in the wild

Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. 2023 · 2023

Earlier work this paper cites.

Instructdet: Diversifying referring object detection with generalized instructions

Original

Ronghao Dang, Jiangyan Feng, Haodong Zhang, Chongjian Ge, Lin Song, Lijun Gong, Chengju Liu, Qijun Chen, Feng Zhu, Rui Zhao, et al. 2023 · 2023

Earlier work this paper cites.

A challenger to gpt-4v? early explorations of gemini in visual expertise

Original

Chaoyou Fu, Renrui Zhang, Haojia Lin, Zihan Wang, Timin Gao, Yongdong Luo, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, et al. 2023 · 2023

Earlier work this paper cites.

Figstep: Jailbreaking large vision-language models via typographic visual prompts

Original

Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. 2023 · 2023

Earlier work this paper cites.

A systematic survey of prompt engineering on vision-language foundation models

Original

Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. 2023 · 2023

Earlier work this paper cites.

Segment anything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023 · 2023

Earlier work this paper cites.

A practical survey on zero-shot prompt design for in-context learning

Original

Yinheng Li. 2023 · 2023

Earlier work this paper cites.

Mm-vid: Advancing video understanding with gpt-4v (ision)

Original

Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, et al. 2023 · 2023

Earlier work this paper cites.

Blackvip: Black-box visual prompting for robust transfer learning

Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. 2023 · 2023

Earlier work this paper cites.

What does clip know about a red circle? visual prompt engineering for vlms

Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023 · 2023

Earlier work this paper cites.

In-context learning unlocked for diffusion models

Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang, Mingyuan Zhou, et al. 2023 · 2023

Earlier work this paper cites.

Few-shot composition learning for image retrieval with prompt tuning

Junda Wu, Rui Wang, Handong Zhao, Ruiyi Zhang, Chaochao Lu, Shuai Li, and Ricardo Henao. 2023 · 2023

Earlier work this paper cites.

Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v

Original

Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023 · 2023

Earlier work this paper cites.

Multimodal prompt perceiver: Empower adaptiveness generalizability and fidelity for all-in-one image restoration

Yuang Ai, Huaibo Huang, Xiaoqiang Zhou, Jiexiang Wang, and Ran He. 2024 · 2024

Cited alongside, same era.

Hallucination of multimodal large language models: A survey

Original

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024 · 2024

Cited alongside, same era.

Vip-llava: Making large multimodal models understand arbitrary visual prompts

Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. 2024 · 2024

Cited alongside, same era.

Controllable generation with text-to-image diffusion models: A survey

Original

Pu Cao, Feng Zhou, Qing Song, and Lu Yang. 2024 · 2024

Cited alongside, same era.

Language-image models with 3d understanding

Original

T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. 2024 · 2024

Closest in time.

Pivot: Iterative visual prompting elicits actionable knowledge for vlms

Original

Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. 2024 · 2024

Closest in time.

Responsible visual editing

Original

Minheng Ni, Yeli Shen, Lei Zhang, and Wangmeng Zuo. 2024 · 2024

Closest in time.

Auto-encoding morph-tokens for multimodal llm

Original

Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, and Hanwang Zhang. 2024 · 2024

Closest in time.

Unified text-to-image generation and retrieval