Fetching the paper…

Reading the bibliography…

2025

VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation

Ma, Wentao, Ren, Weiming, Jia, Yiming et al.

Original
PDF
Post on X

Understand

Large multimodal models (LMMs) have recently emerged as a powerful tool for long video understanding (LVU), prompting the development of standardized LVU benchmarks to evaluate their performance.

However, our investigation reveals a rather sober lesson for existing LVU benchmarks.
First, most existing benchmarks rely heavily on multiple-choice questions (MCQs), whose evaluation results are inflated due to the possibility of guessing the correct answer; Second, a significant portion of questions in these benchmarks have strong priors to allow models to answer directly without even reading the input video.
For example, Gemini-1.5-Pro can achieve over 50\% accuracy given a random frame from a long video on Video-MME.

VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation · Around

Built on

Video question answering via gradually refined attention over appearance and motion
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang · 2017
Earlier work this paper cites.
Towards automatic learning of procedures from web instructional videos
Luowei Zhou, Chenliang Xu, and Jason Corso · 2018
Earlier work this paper cites.
Activitynet-qa: A dataset for understanding complex web videos via question answering
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao · 2019
Earlier work this paper cites.
Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al · 2021
Earlier work this paper cites.
Video-llava: Learning united visual representation by alignment before projection
Original
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan · 2023
Earlier work this paper cites.
Video-chatgpt: Towards detailed video understanding via large vision and language models
Original
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan · 2023
Earlier work this paper cites.
Video-llama: An instruction-tuned audio-visual language model for video understanding
Original
Hang Zhang, Xin Li, and Lidong Bing · 2023
Earlier work this paper cites.
Mamba: Linear-time sequence modeling with selective state spaces
Original
Albert Gu and Tri Dao · 2023
Earlier work this paper cites.
Judging llm-as-a-judge with mt-bench and chatbot arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al · 2023
Earlier work this paper cites.
Flashattention-2: Faster attention with better parallelism and work partitioning
Original
Tri Dao · 2023
Earlier work this paper cites.
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis
Original
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al · 2024
Earlier work this paper cites.
Video anomaly detection and explanation via large language models
Original
Hui Lv and Qianru Sun · 2024
Earlier work this paper cites.
Vista: Enhancing long-duration and high-resolution video understanding by video spatiotemporal augmentation
Original
Weiming Ren, Huan Yang, Jie Min, Cong Wei, and Wenhu Chen · 2024
Earlier work this paper cites.

Similar

Mlvu: A comprehensive benchmark for multi-task long video understanding
Original
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu · 2024
Cited alongside, same era.
Longvideobench: A benchmark for long-context interleaved video-language understanding
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li · 2024
Cited alongside, same era.
Hourvideo: 1-hour video-language understanding
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li · 2024
Cited alongside, same era.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Original
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al · 2024
Cited alongside, same era.
Mantis: Interleaved multi-image instruction tuning

Then

Internvideo2. 5: Empowering video mllms with long and rich context modeling
Original
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al · 2025
Closest in time.
Vamba: Understanding hour-long videos with hybrid mamba-transformers
Original
Weiming Ren, Wentao Ma, Huan Yang, Cong Wei, Ge Zhang, and Wenhu Chen · 2025
Closest in time.
Token-efficient long video understanding for multimodal llms
Original
Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, et al · 2025
Closest in time.
Bimba: Selective-scan compression for long-range video question answering
Original
Md Mohaiminul Islam, Tushar Nagarajan, Huiyu Wang, Gedas Bertasius, and Lorenzo Torresani · 2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen · 2024

Cited alongside, same era.

Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos

Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang · 2024

Cited alongside, same era.

Longvu: Spatiotemporal adaptive compression for long video-language understanding

Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al · 2024

Cited alongside, same era.

Video-xl: Extra-long vision language model for hour-scale video understanding

Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao · 2024

Cited alongside, same era.

Vript: A video is worth thousands of words

Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao · 2024

Cited alongside, same era.

Tempcompass: Do video llms really understand videos?

Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou · 2024

Cited alongside, same era.

Measuring short-form factuality in large language models

Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus · 2024

Cited alongside, same era.

A survey on llm-as-a-judge

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al · 2024

Cited alongside, same era.

Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning

Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang · 2025

Closest in time.

Video-r1: Reinforcing video reasoning in mllms

Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue · 2025

Closest in time.

Video-xl-pro: Reconstructive token compression for extremely long video understanding

Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao · 2025

Closest in time.

Qwen2. 5-vl technical report

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al · 2025

Closest in time.

Video simpleqa: Towards factuality evaluation in large video language models

Meng Cao, Pengfei Hu, Yingyao Wang, Jihao Gu, Haoran Tang, Haoze Zhao, Jiahua Dong, Wangbo Yu, Ge Zhang, Ian Reid, et al · 2025

Closest in time.

Gpt-4o mini: Advancing cost-efficient intelligence, July 2024b

OpenAI · 2025

Closest in time.

Introducing gpt-4.1 in the api, April 2025

OpenAI · 2025

Closest in time.

Gemini 2.5: Our most intelligent ai model, March 2025

Koray Kavukcuoglu · 2025

Closest in time.

Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al · 2025

Closest in time.