Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model

Vqa: Visual question answering

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015 · 2015

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Original

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Earlier work this paper cites.

Flamingo: a visual language model for few-shot learning

Original

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022 · 2022

Earlier work this paper cites.

Chartqa: A benchmark for question answering about charts with visual and logical reasoning

Original

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022 · 2022

Earlier work this paper cites.

Multi-view reasoning: Consistent contrastive learning for math word problem

Wenqi Zhang, Yongliang Shen, Yanna Ma, Xiaoxia Cheng, Zeqi Tan, Qingpeng Nong, and Weiming Lu. 2022 · 2022

Earlier work this paper cites.

Improving image generation with better captions

James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023 · 2023

Earlier work this paper cites.

Palm-e: An embodied multimodal language model

Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. 2023 · 2023

Earlier work this paper cites.

What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning

Original

Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, and Ji-Rong Wen. 2023 · 2023

Earlier work this paper cites.

Mme: A comprehensive evaluation benchmark for multimodal large language models

Original

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023 · 2023

Earlier work this paper cites.

Chartllama: A multimodal llm for chart understanding and generation

Original

Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. 2023 · 2023

Earlier work this paper cites.

Cogagent: A visual language model for gui agents

Original

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2023 · 2023

Earlier work this paper cites.

mplug-paperowl: Scientific diagram analysis with the multimodal large language model

Original

Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan, Chenliang Li, Qi Qian, Ji Zhang, and Fei Huang. 2023 · 2023

Earlier work this paper cites.

Language is not all you need: Aligning perception with language models

Original

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. 2023 · 2023

Earlier work this paper cites.

Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

Original

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023 · 2023

Earlier work this paper cites.

Unichart: A universal vision-language pretrained model for chart comprehension and reasoning

Original

Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq R. Joty. 2023 · 2023

Earlier work this paper cites.

Gpt-4 technical report

OpenAI. 2023 · 2023

Earlier work this paper cites.

Kosmos-2: Grounding multimodal large language models to the world

Original

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023 · 2023

Earlier work this paper cites.

Pandagpt: One model to instruction-follow them all

Original

Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023 · 2023

Earlier work this paper cites.

Gemini: a family of highly capable multimodal models

Original

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023 · 2023

Earlier work this paper cites.

Finvis-gpt: A multimodal large language model for financial chart analysis

Original

Ziao Wang, Yuhang Li, Junda Wu, Jaehyeon Soon, and Xiaofeng Zhang. 2023 · 2023

Cited alongside, same era.

Magicoder: Source code is all you need

Original

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023 · 2023

Cited alongside, same era.

Structchart: Perception, structuring, reasoning for visual chart understanding

Original

Renqiu Xia, Bo Zhang, Hao Peng, Ning Liao, Peng Ye, Botian Shi, Junchi Yan, and Y. Qiao. 2023 · 2023

Cited alongside, same era.

A survey on multimodal large language models

Original

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023 · 2023

Cited alongside, same era.

Glm-130b: An Open Bilingual Pre-trained Model

Llm2llm: Boosting llms with novel iterative data enhancement

Original

Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. 2024 · 2024

Closest in time.

Autocoder: Enhancing code large language model with {AIEV-Instruct}

Bin Lei, Yuchen Li, and Qiuwu Chen. 2024 · 2024

Closest in time.

Seed-bench: Benchmarking multimodal large language models

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024 · 2024

Closest in time.

Deepseek-vl: towards real-world vision-language understanding

Original

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. 2024 · 2024

Closest in time.

Chartinstruct: Instruction tuning for chart comprehension and reasoning

Original