LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

Towards vqa models that can read

Original

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019a · 1904

Earlier work this paper cites.

Scene text visual question answering

Original

Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusiñol, Ernest Valveny, C. V. Jawahar, and Dimosthenis Karatzas. 2019 · 1905

Earlier work this paper cites.

Coresets for data-efficient training of machine learning models

Original

Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. 2020 · 1906

Earlier work this paper cites.

Taxonomy of

Benjamin S Bloom et al. 1956 · 1956

Earlier work this paper cites.

Statistical theories of mental test scores

Frederic M Lord, Melvin R Novick, and Allan Birnbaum. 1968 · 1968

Earlier work this paper cites.

Least squares quantization in pcm

Stuart Lloyd. 1982 · 1982

Earlier work this paper cites.

Combinatorial Optimization

W. Cook. 1997 · 1997

Earlier work this paper cites.

Textcaps: a dataset for image captioning with reading comprehension

Original

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020a · 2003

Earlier work this paper cites.

Language models are few-shot learners

Original

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2005

Earlier work this paper cites.

Docvqa: A dataset for vqa on document images. corr abs/2007.00398 (2020)

Original

Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. 2020 · 2007

Earlier work this paper cites.

Active learning by querying informative and representative examples

Sheng-jun Huang, Rong Jin, and Zhi-Hua Zhou. 2010 · 2010

Earlier work this paper cites.

ReferItGame: Referring to objects in photographs of natural scenes

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014 · 2014

Earlier work this paper cites.

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014 · 2014

Earlier work this paper cites.

A diagram is worth a dozen images

Original

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016 · 2016

Earlier work this paper cites.

Generation and comprehension of unambiguous object descriptions

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016 · 2016

Earlier work this paper cites.

Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?

Original

Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. 2024d · 2016

Earlier work this paper cites.

Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017 · 2017

Earlier work this paper cites.

Vizwiz grand challenge: Answering visual questions from blind people

Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018 · 2018

Earlier work this paper cites.

Active learning for convolutional neural networks: A core-set approach

Ozan Sener and Silvio Savarese. 2018 · 2018

Earlier work this paper cites.

nocaps: novel object captioning at scale

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019 · 2019

Earlier work this paper cites.

Gpipe: Efficient training of giant neural networks using pipeline parallelism

Original

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019 · 2019

Earlier work this paper cites.

Gqa: A new dataset for real-world visual reasoning and compositional question answering

Drew A Hudson and Christopher D Manning. 2019 · 2019

Earlier work this paper cites.

Understanding black-box predictions via influence functions

Original

Pang Wei Koh and Percy Liang. 2020 · 2020

Earlier work this paper cites.

Websrc: A dataset for web-based structural reading comprehension

Original

Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021 · 2021

Cited alongside, same era.

Learning transferable visual models from natural language supervision

Original

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021 · 2021

Cited alongside, same era.

Ocr-free document understanding transformer

Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022 · 2022

Cited alongside, same era.

Holistic evaluation of language models

Original

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022 · 2022

Cited alongside, same era.

Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

Original

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023 · 2023

Later among the works it cites.

Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition

Original

Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. 2023 · 2023

Later among the works it cites.

Introducing the next generation of claude

Anthropic. 2024 · 2024

Closest in time.

Seeclick: Harnessing gui grounding for advanced visual gui agents

Original

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024 · 2024

Closest in time.