Built on

Logiqa: A challenge dataset for machine reading comprehension with logical reasoning
Original
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020 · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021 · 2021
Earlier work this paper cites.
Revive: Regional visual representation matters in knowledge-based visual question answering
Original
Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. 2022 · 2022
Earlier work this paper cites.
Towards end-to-end embodied decision making via multi-modal large language model: Explorations with gpt4-vision and beyond
Original
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu Liu, and Baobao Chang. 2023 · 2023
Earlier work this paper cites.

Similar

Palm-e: An embodied multimodal language model
Original
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. 2023 · 2023
Cited alongside, same era.
Promptcap: Prompt-guided task-aware image captioning
Original
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2023 · 2023
Cited alongside, same era.
Chatgpt for shaping the future of dentistry: the potential of multi-modal large language model
Hanyao Huang, Ou Zheng, Dongdong Wang, Jiayi Yin, Zijin Wang, Shengxuan Ding, Heng Yin, Chuan Xu, Renjie Yang, Qian Zheng, et al. 2023 · 2023
Cited alongside, same era.
Exploring ocr capabilities of gpt-4v(ision) : A quantitative and in-depth evaluation
Original
Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, and Lianwen Jin. 2023 · 2023
Cited alongside, same era.

Then

Can gpt-4v(ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis
Original
Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023 · 2023
Later among the works it cites.
The dawn of lmms: Preliminary explorations with gpt-4v (ision)
Original
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023 · 2023
Later among the works it cites.
Ttida: Controllable generative data augmentation via text-to-text and text-to-image models
Original
Yuwei Yin, Jean Kaddour, Xiang Zhang, Yixin Nie, Zhenguang Liu, Lingpeng Kong, and Qi Liu. 2023 · 2023
Later among the works it cites.
Mm-r 3: On (in-) consistency of multi-modal large language models (mllms)
Original
Shih-Han Chou, Shivam Chandhok, James J Little, and Leonid Sigal. 2024 · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Cross-Modal Consistency in Multimodal Large Language Models

Zhang, Xiang, Li, Senyu, Shi, Ning et al.

Understand

Recent developments in multimodal methodologies have marked the beginning of an exciting era for models adept at processing diverse data types, encompassing text, audio, and visual content.

Models like GPT-4V, which merge computer vision with advanced language processing, exhibit extraordinary proficiency in handling intricate tasks that require a simultaneous understanding of both textual and visual information.
Prior research efforts have meticulously evaluated the efficacy of these Vision Large Language Models (VLLMs) in various domains, including object detection, image captioning, and other related fields.
However, existing analyses have often suffered from limitations, primarily centering on the isolated evaluation of each modality's performance while neglecting to explore their intricate cross-modal interactions.

Built on

Logiqa: A challenge dataset for machine reading comprehension with logical reasoning
Original
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020 · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021 · 2021
Earlier work this paper cites.
Revive: Regional visual representation matters in knowledge-based visual question answering
Original
Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. 2022 · 2022
Earlier work this paper cites.
Towards end-to-end embodied decision making via multi-modal large language model: Explorations with gpt4-vision and beyond
Original
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu Liu, and Baobao Chang. 2023 · 2023
Earlier work this paper cites.

Similar

Palm-e: An embodied multimodal language model
Original
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. 2023 · 2023
Cited alongside, same era.
Promptcap: Prompt-guided task-aware image captioning
Original
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2023 · 2023
Cited alongside, same era.
Chatgpt for shaping the future of dentistry: the potential of multi-modal large language model
Hanyao Huang, Ou Zheng, Dongdong Wang, Jiayi Yin, Zijin Wang, Shengxuan Ding, Heng Yin, Chuan Xu, Renjie Yang, Qian Zheng, et al. 2023 · 2023
Cited alongside, same era.
Exploring ocr capabilities of gpt-4v(ision) : A quantitative and in-depth evaluation
Original
Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, and Lianwen Jin. 2023 · 2023
Cited alongside, same era.

Then

Can gpt-4v(ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis
Original
Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023 · 2023
Later among the works it cites.
The dawn of lmms: Preliminary explorations with gpt-4v (ision)
Original
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023 · 2023
Later among the works it cites.
Ttida: Controllable generative data augmentation via text-to-text and text-to-image models
Original
Yuwei Yin, Jean Kaddour, Xiang Zhang, Yixin Nie, Zhenguang Liu, Lingpeng Kong, and Qi Liu. 2023 · 2023
Later among the works it cites.
Mm-r 3: On (in-) consistency of multi-modal large language models (mllms)
Original
Shih-Han Chou, Shivam Chandhok, James J Little, and Leonid Sigal. 2024 · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

On the road with gpt-4v(ision): Early explorations of visual-language model on autonomous driving

Original

Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, Zheng Zhu, Shaoyan Sun, Yeqi Bai, Xinyu Cai, Min Dou, Shuanglu Hu, Botian Shi, and Yu Qiao. 2023 · 2023

Cited alongside, same era.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a

Cited in the paper.

Measuring mathematical problem solving with the math dataset

Original

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b

Cited in the paper.

Mmc: Advancing multimodal chart understanding with large-scale instruction tuning

Original

Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. 2023a

Cited in the paper.

Holistic evaluation of gpt-4v for biomedical imaging

Original

Zhengliang Liu, Hanqi Jiang, Tianyang Zhong, Zihao Wu, Chong Ma, Yiwei Li, Xiaowei Yu, Yutong Zhang, Yi Pan, Peng Shu, Yanjun Lyu, Lu Zhang, Junjie Yao, Peixin Dong, Chao Cao, Zhenxiang Xiao, Jiaqi Wang, Huan Zhao, Shaochen Xu, Yaonai Wei, Jingyuan Chen, Haixing Dai, Peilong Wang, Hao He, Zewei Wang, Xinyu Wang, Xu Zhang, Lin Zhao, Yiheng Liu, Kai Zhang, Liheng Yan, Lichao Sun, Jun Liu, Ning Qiang, Bao Ge, Xiaoyan Cai, Shijie Zhao, Xintao Hu, Yixuan Yuan, Gang Li, Shu Zhang, Xin Zhang, Xi Jiang, Tuo Zhang, Dinggang Shen, Quanzheng Li, Wei Liu, Xiang Li, Dajiang Zhu, and Tianming Liu. 2023b

Cited in the paper.

Vision-language models for vision tasks: A survey

Original

Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2023a

Cited in the paper.

Don’t trust gpt when your question is not in english

Original

Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023b

Cited in the paper.

Bridging the gap between BabelNet and HowNet: Unsupervised sense alignment and sememe prediction

Xiang Zhang, Ning Shi, Bradley Hauer, and Grzegorz Kondrak. 2023c

Cited in the paper.

Contranovo: A contrastive learning approach to enhance de novo peptide sequencing

Zhi Jin, Sheng Xu, Xiang Zhang, Tianze Ling, Nanqing Dong, Wanli Ouyang, Zhiqiang Gao, Cheng Chang, and Siqi Sun. 2024 · 2024

Closest in time.