MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale

Icdar 2019 robust reading challenge on reading chinese text on signboard

Original

Xi Liu, Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, et al. 2019 · 1912

Earlier work this paper cites.

All you may need for VQA are image captions

Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. 2022 · 1963

Earlier work this paper cites.

The iam-database: an english sentence database for offline handwriting recognition

Urs-Viktor Marti and Horst Bunke. 2002 · 2002

Earlier work this paper cites.

Pathvqa: 30000+ questions for medical visual question answering

Original

Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020 · 2003

Earlier work this paper cites.

Docvqa: A dataset for vqa on document images. corr abs/2007.00398 (2020)

Original

Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. 2020 · 2007

Earlier work this paper cites.

Interrater reliability: the kappa statistic

Mary L McHugh. 2012 · 2012

Earlier work this paper cites.

Scene text recognition using higher order language priors

Anand Mishra, Karteek Alahari, and C. V. Jawahar. 2012 · 2012

Earlier work this paper cites.

VQA: visual question answering

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015 · 2015

Earlier work this paper cites.

Microsoft coco: Common objects in context

Original

Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015 · 2015

Earlier work this paper cites.

Exploring models and data for image question answering

Original

Mengye Ren, Ryan Kiros, and Richard Zemel. 2015 · 2015

Earlier work this paper cites.

Imagenet large scale visual recognition challenge

Original

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015 · 2015

Earlier work this paper cites.

Solving geometry problems: Combining text and diagram interpretation

Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015 · 2015

Earlier work this paper cites.

Visual storytelling

Original

Ting-Hao K. Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Aishwarya Agrawal, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016 · 2016

Earlier work this paper cites.

A diagram is worth a dozen images

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016 · 2016

Earlier work this paper cites.

Modeling context in referring expressions

Original

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016 · 2016

Earlier work this paper cites.

Visual7w: Grounded question answering in images

Yuke Zhu, Oliver Groth, Michael S. Bernstein, and Li Fei-Fei. 2016 · 2016

Earlier work this paper cites.

Making the v in vqa matter: Elevating the role of image understanding in visual question answering

Original

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017 · 2017

Earlier work this paper cites.

Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

Original

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017 · 2017

Earlier work this paper cites.

Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension

Aniruddha Kembhavi, Min Joon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017 · 2017

Earlier work this paper cites.

Visual genome: Connecting language and vision using crowdsourced dense image annotations

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, and Li Fei-Fei. 2017 · 2017

Earlier work this paper cites.

Icdar2017 competition on reading chinese text in the wild (rctw-17)

Original

Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai. 2017 · 2017

Earlier work this paper cites.

Imagine this! scripts to compositions to videos

Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. 2018 · 2018

Earlier work this paper cites.

Vizwiz grand challenge: Answering visual questions from blind people

Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018 · 2018

Earlier work this paper cites.

DVQA: understanding data visualizations via question answering

Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. 2018 · 2018

Earlier work this paper cites.

Figureqa: An annotated figure dataset for visual reasoning

Original

Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. 2018 · 2018

Earlier work this paper cites.

A dataset of clinically generated visual questions and answers about radiology images

Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018 · 2018

Earlier work this paper cites.

Tallyqa: Answering complex counting questions

Original

Manoj Acharya, Kushal Kafle, and Christopher Kanan. 2019 · 2019

Earlier work this paper cites.

Scene text visual question answering

Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluís Gómez i Bigorda, Marçal Rusiñol, C. V. Jawahar, Ernest Valveny, and Dimosthenis Karatzas. 2019 · 2019

Earlier work this paper cites.

Icdar2019 competition on scanned receipt ocr and information extraction

Original

Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. 2019 · 2019

Earlier work this paper cites.

Gqa: A new dataset for real-world visual reasoning and compositional question answering

Original

Drew A Hudson and Christopher D Manning. 2019 · 2019

Earlier work this paper cites.

Storygan: A sequential conditional GAN for story visualization

Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David E. Carlson, and Jianfeng Gao. 2019 · 2019

Earlier work this paper cites.

Ocr-vqa: Visual question answering by reading text in images

Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019 · 2019

Earlier work this paper cites.

Towards VQA models that can read

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019 · 2019

Earlier work this paper cites.

A corpus for reasoning about natural language grounded in photographs

Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019 · 2019

Earlier work this paper cites.

RAVEN: A dataset for relational and analogical visual reasoning

Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019 · 2019

Earlier work this paper cites.

The hateful memes challenge: Detecting hate speech in multimodal memes

Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020 · 2020

Earlier work this paper cites.

Plotqa: Reasoning over scientific plots

Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020 · 2020

Earlier work this paper cites.

Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model

Jason Obeid and Enamul Hoque. 2020 · 2020

Earlier work this paper cites.

Connecting vision and language with localized narratives

Original

Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020 · 2020

Cited alongside, same era.

Textcaps: a dataset for image captioningwith reading comprehension

Original

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020 · 2020

Cited alongside, same era.

GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. 2021a · 2021

Cited alongside, same era.

FinQA: A dataset of numerical reasoning over financial data

Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021b · 2021

Cited alongside, same era.

Docvqa: A dataset for vqa on document images

Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021b · 2021

VisText: A Benchmark for Semantically Rich Chart Captioning

Benny J. Tang, Angie Boggust, and Arvind Satyanarayan. 2023 · 2023

Later among the works it cites.

Llama: Open and efficient foundation language models

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 · 2023

Later among the works it cites.

How many unicorns are in this image? a safety evaluation benchmark for vision llms

Original

Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. 2023 · 2023

Later among the works it cites.

To see is to believe: Prompting gpt-4v for better visual instruction tuning

Original

Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. 2023 · 2023

Later among the works it cites.