Fetching the paper…
Reading the bibliography…
With the rapid advancements in multi-modal large language models (MLLMs), connectors play a pivotal role in bridging diverse modalities and enhancing model performance.
Flamingo: a visual language model for few-shot learning
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al · 2022
Earlier work this paper cites.
Minigpt-v2: large language model as a unified interface for vision-language multi-task learning
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny · 2023
Earlier work this paper cites.
From clip to dino: Visual encoders shout in multi-modal large language models
Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin’e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong · 2023
Earlier work this paper cites.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi · 2023
Earlier work this paper cites.
Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration
Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu · 2023
Earlier work this paper cites.
How to bridge the gap between modalities: A comprehensive survey on multimodal large language model
Shezheng Song, Xiaopeng Li, Shasha Li, Shan Zhao, Jie Yu, Jun Ma, Xiaoguang Mao, and Weimin Zhang · 2023
Earlier work this paper cites.
Pandagpt: One model to instruction-follow them all
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai · 2023
Earlier work this paper cites.
Voila-a: Aligning vision-language models with user’s gaze attention
Kun Yan, Lei Ji, Zeyu Wang, Yuntao Wang, Nan Duan, and Shuai Ma · 2023
Earlier work this paper cites.
A survey on multimodal large language models
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen · 2023
Earlier work this paper cites.
Minigpt-4: Enhancing vision-language understanding with advanced large language models
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny · 2023
Earlier work this paper cites.
Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji · 2024
Earlier work this paper cites.
Mmfuser: Multimodal multi-layer feature fuser for fine-grained vision-language understanding
Yue Cao, Yangzhou Liu, Zhe Chen, Guangchen Shi, Wenhai Wang, Danhuai Zhao, and Tong Lu · 2024
Earlier work this paper cites.
Honeybee: Locality-enhanced projector for multimodal llm
Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh · 2024
Earlier work this paper cites.
Lion: Empowering multimodal large language model with dual-level visual knowledge
Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie · 2024
Earlier work this paper cites.
How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al · 2024
Earlier work this paper cites.
Surgfc: Multimodal surgical function calling framework on the demand of surgeons
Zhen Chen, Xingjian Luo, Jinlin Wu, Danny TM Chan, Zhen Lei, Sebastien Ourselin, and Hongbin Liu · 2024
Earlier work this paper cites.
Meerkat: Audio-visual large language model for grounding in space and time
Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha · 2024
Earlier work this paper cites.
Query-based cross-modal projector bolstering mamba multimodal llm
SooHwan Eom, Jay Shim, Gwanhyeong Koo, Haebin Na, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo · 2024
Earlier work this paper cites.
Vitron: A unified pixel-level vision llm for understanding, generating, segmenting, editing
Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan · 2024
Cited alongside, same era.
Llava-ultra: Large chinese language and vision assistant for ultrasound
Xuechen Guo, Wenhao Chai, Shi-Yan Li, and Gaoang Wang · 2024
Cited alongside, same era.
Onellm: One framework to align all modalities with language
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue · 2024
Cited alongside, same era.
Mmneuron: Discovering neuron-level domain-specific interpretation in multimodal large language model
Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, and Xuming Hu · 2024
Cited alongside, same era.
Tokenpacker: Efficient visual projector for multimodal llm
Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang · 2024
V ∗ : Guided visual search as a core mechanism in multimodal llms
Penghao Wu and Saining Xie · 2024
Later among the works it cites.
Pllava: Parameter-free llava extension from images to videos for video dense captioning
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng · 2024
Later among the works it cites.
Chartmoe: Mixture of expert connector for advanced chart understanding
Zhengzhuo Xu, Bowen Qu, Yiyan Qi, Sinan Du, Chengjin Xu, Chun Yuan, and Jian Guo · 2024
Later among the works it cites.
Planllm: Video procedure planning with refinable large language models
Dejie Yang, Zijing Zhao, et al · 2024
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
3dmit: 3d multi-modal instruction tuning for scene understanding
Zeju Li, Chao Zhang, Xiaoyan Wang, Ruilong Ren, Yifan Xu, Ruifei Ma, Xiangde Liu, and Rong Wei · 2024
Cited alongside, same era.
Groundinggpt: Language enhanced multi-modal grounding model
Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Vu Tu, et al · 2024
Cited alongside, same era.
Infimm: Advancing multimodal understanding with an open-sourced visual language model
Haogeng Liu, Quanzeng You, Yiqi Wang, Xiaotian Han, Bohan Zhai, Yongfei Liu, Wentao Chen, Yiren Jian, Yunzhe Tao, Jianbo Yuan, et al · 2024
Cited alongside, same era.
Improved baselines with visual instruction tuning
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee · 2024
Cited alongside, same era.
Visual instruction tuning
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee · 2024
Cited alongside, same era.
Ppllava: Varied video sequence understanding with prompt guidance
Ruyang Liu, Haoran Tang, Haibo Liu, Yixiao Ge, Ying Shan, Chen Li, and Jiankun Yang · 2024
Cited alongside, same era.
Anymal: An efficient and scalable any-modality augmented language model
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, et al · 2024
Cited alongside, same era.
Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang · 2024
Later among the works it cites.
Deco: Decoupling token compression from semantic abstraction in multimodal large language models
Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou · 2024
Later among the works it cites.
mplug-owl3: Towards long image-sequence understanding in multi-modal large language models
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou · 2024
Later among the works it cites.
Yi: Open foundation models by 01. ai
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al · 2024
Later among the works it cites.
World knowledge-enhanced reasoning using instruction-guided interactor in autonomous driving
Mingliang Zhai, Cheng Li, Zengyuan Guo, Ningrui Yang, Xiameng Qin, Yuwei Wu, Sanyuan Zhao, Junyu Han, Ji Tao, and Yunde Jia · 2024
Later among the works it cites.
Mm-llms: Recent advances in multimodal large language models
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu · 2024
Later among the works it cites.
Dockylin: A large multimodal model for visual document understanding with efficient visual slimming
Jiaxin Zhang, Wentao Yang, Songxuan Lai, Zecheng Xie, and Lianwen Jin · 2024
Later among the works it cites.
Uni-med: A unified medical generalist foundation model for multi-task learning via connector-moe
Xun Zhu, Ying Hu, Fanbin Mo, Miao Li, and Ji Wu · 2024
Later among the works it cites.
Focusllava: A coarse-to-fine approach for efficient and effective visual token compression
Yuke Zhu, Chi Xie, Shuang Liang, Bo Zheng, and Sheng Guo · 2024
Later among the works it cites.
Brave: Broadening the visual encoding of vision-language models
Oğuzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari · 2025
Closest in time.
Cumo: Scaling multimodal llm with co-upcycled mixture-of-experts
Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen · 2025
Closest in time.
Mm1: methods, analysis and insights from multimodal llm pre-training
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al · 2025
Closest in time.
Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios
Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao · 2025
Closest in time.