LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating

Unifying vision-and-language tasks via text generation

Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021 · 1942

Earlier work this paper cites.

LayoutLM: Pre-training of text and layout for document image understanding

Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020 · 2020

Earlier work this paper cites.

Docformer: End-to-end transformer for document understanding

Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. 2021 · 2021

Earlier work this paper cites.

Docvqa: A dataset for VQA on document images

Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021 · 2021

Earlier work this paper cites.

LayoutLMv2: Multi-modal pre-training for visually-rich document understanding

Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021 · 2021

Earlier work this paper cites.

Layoutlmv3: Pre-training for document AI with unified text and image masking

Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022 · 2022

Earlier work this paper cites.

Chartqa: A benchmark for question answering about charts with visual and logical reasoning

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. 2022 · 2022

Earlier work this paper cites.

Hierarchical multimodal transformers for multi-page docvqa

Original

Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. 2022 · 2022

Earlier work this paper cites.

Document understanding dataset and evaluation (DUDE)

Jordy Van Landeghem, Rafal Powalski, Rubèn Tito, Dawid Jurkiewicz, Matthew B. Blaschko, Lukasz Borchmann, Mickaël Coustaty, Sien Moens, Michal Pietruszka, Bertrand Anckaert, Tomasz Stanislawek, Pawel Józiak, and Ernest Valveny. 2023 · 2023

Earlier work this paper cites.

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 · 2023

Earlier work this paper cites.

Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution

Original

Alibaba. 2024 · 2024

Earlier work this paper cites.

Claude 3.5 sonnet

Anthropic. 2024 · 2024

Cited alongside, same era.

GRAM: global reasoning for multi-page VQA

Tsachi Blau, Sharon Fogel, Roi Ronen, Alona Golts, Shahar Tsiper, Elad Ben-Avraham, Aviad Aberdam, Roy Ganz, and Ron Litman. 2024 · 2024

Cited alongside, same era.

M-longdoc: A benchmark for multimodal super-long document understanding and a retrieval-aware tuning framework

Original

Yew Ken Chia, Liying Cheng, Hou Pong Chan, Chaoqun Liu, Maojia Song, Sharifah Mahani Aljunied, Soujanya Poria, and Lidong Bing. 2024 · 2024

Cited alongside, same era.

M3docrag: Multi-modal retrieval is what you need for multi-page multi-document understanding

Original

Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. 2024 · 2024

Cited alongside, same era.

MMVQA: A comprehensive dataset for investigating multipage multimodal information retrieval in pdf-based visual question answering

Yihao Ding, Kaixuan Ren, Jiabin Huang, Siwen Luo, and Soyeon Caren Han. 2024a · 2024

Multi-page document visual question answering using self-attention scoring mechanism

Lei Kang, Rubèn Tito, Ernest Valveny, and Dimosthenis Karatzas. 2024 · 2024

Closest in time.

Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

Original

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024 · 2024

Closest in time.

TextMonkey: An ocr-free large multimodal model for understanding document

Original

Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024 · 2024

Closest in time.

Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024 · 2024

Closest in time.

Mmlongbench-doc: Benchmarking long-context document understanding with visualizations