2022

Video Question Answering: Datasets, Algorithms and Challenges

Zhong, Yaoyao, Xiao, Junbin, Ji, Wei et al.

Understand

Video Question Answering (VideoQA) aims to answer natural language questions according to the given videos.

It has earned increasing attention with recent research trends in joint vision and language understanding.
Yet, compared with ImageQA, VideoQA is largely underexplored and progresses slowly.
Although different algorithms have continually been proposed and shown success on different VideoQA datasets, we find that there lacks a meaningful survey to categorize them, which seriously impedes its advancements.

Built on

Video question answering: a survey of models and datasets
Guanglu Sun, Lili Liang, Tianlin Li, Bo Yu, et al. 2021 · 1937
Earlier work this paper cites.
Verb semantics and lexical selection
Zhibiao Wu and Martha Stone Palmer. 1994 · 1994
Earlier work this paper cites.
Heterogeneous memory enhanced multimodal attention model for video question answering
Chenyou Fan, Xiaofan Zhang, Shu Zhang, et al. 2019 · 2007
Earlier work this paper cites.
iperceive: Applying common-sense reasoning to multi-modal dense video captioning and video question answering
Original
Aman Chadha, Gurneet Arora, and Navpreet Kaloty. 2020 · 2011
Earlier work this paper cites.
Vqa: Visual question answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015 · 2015
Earlier work this paper cites.
End-to-end memory networks
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015 · 2015
Earlier work this paper cites.
Hierarchical question-image co-attention for visual question answering
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016 · 2016
Earlier work this paper cites.
Movieqa: Understanding stories in movies through question-answering
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016 · 2016
Earlier work this paper cites.
Msr-vtt: A large video description dataset for bridging video and language
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016 · 2016
Earlier work this paper cites.
Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. 2017 · 2017
Earlier work this paper cites.
Tgif-qa: Toward spatio-temporal reasoning in visual question answering
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017 · 2017
Earlier work this paper cites.
Semi-supervised classification with graph convolutional networks
Thomas N Kipf and Max Welling. 2017 · 2017
Earlier work this paper cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017 · 2017
Earlier work this paper cites.
A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering
Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. 2017 · 2017
Earlier work this paper cites.
Marioqa: Answering questions by watching gameplay videos
Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han. 2017 · 2017
Earlier work this paper cites.
A read-write memory network for movie story understanding
Seil Na, Sangho Lee, Jisung Kim, and Gunhee Kim. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Video question answering via gradually refined attention over appearance and motion
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017 · 2017
Earlier work this paper cites.
Leveraging video descriptions to learn video question answering
Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. 2017 · 2017
Earlier work this paper cites.
Uncovering the temporal context for video question answering
Linchao Zhu, Zhongwen Xu, Yi Yang, and Alexander G Hauptmann. 2017 · 2017
Earlier work this paper cites.
Bottom-up and top-down attention for image captioning and visual question answering
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018 · 2018
Earlier work this paper cites.
Temporally grounding natural sentence in video
Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. 2018 · 2018
Earlier work this paper cites.
Motion-appearance co-memory networks for video question answering
Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. 2018 · 2018
Earlier work this paper cites.
Tvqa: Localized, compositional video question answering
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018 · 2018
Earlier work this paper cites.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018 · 2018
Earlier work this paper cites.
Explore multi-step reasoning in video question answering
Xiaomeng Song, Yucheng Shi, Xin Chen, and Yahong Han. 2018 · 2018
Earlier work this paper cites.
A better way to attend: Attention with trees for video question answering
Hongyang Xue, Wenqing Chu, Zhou Zhao, and Deng Cai. 2018 · 2018
Earlier work this paper cites.
Neural-symbolic vqa: Disentangling reasoning from vision and language understanding
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018 · 2018
Earlier work this paper cites.
Open-ended long-form video question answering via adaptive hierarchical reinforced networks
Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhou Yu, Jun Yu, Deng Cai, Fei Wu, and Yueting Zhuang. 2018 · 2018
Earlier work this paper cites.
Egovqa-an egocentric video question answering benchmark dataset
Chenyou Fan. 2019 · 2019
Earlier work this paper cites.
Learning to compose and reason with language tree structures for visual grounding
Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, and Hanwang Zhang. 2019 · 2019
Earlier work this paper cites.
Video question answering with spatio-temporal reasoning
Yunseok Jang, Yale Song, Chris Dongjoo Kim, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2019 · 2019
Earlier work this paper cites.
Video question answering via knowledge-based progressive spatial-temporal attention network
Weike Jin, Zhou Zhao, Yimeng Li, Jie Li, Jun Xiao, and Yueting Zhuang. 2019 · 2019
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
Progressive attention memory network for movie story question answering
Junyeong Kim, Minuk Ma, Kyungsu Kim, Sungjin Kim, et al. 2019 · 2019
Earlier work this paper cites.

Similar

Beyond rnns: Positional self-attention with co-attention for video question answering
Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, et al. 2019 · 2019
Cited alongside, same era.
Multimodal dialog system: Generating responses via adaptive decoders
Liqiang Nie, Wenjie Wang, Richang Hong, Meng Wang, and Qi Tian. 2019 · 2019
Cited alongside, same era.
Annotating objects and relations in user-generated videos
Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. 2019 · 2019
Cited alongside, same era.
Activitynet-qa: A dataset for understanding complex web videos via question answering
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019 · 2019
Cited alongside, same era.
Social-iq: A question answering benchmark for artificial social intelligence
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. 2019 · 2019
Cited alongside, same era.
Lifeqa: A real-life dataset for video question answering
Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang, Jia Deng, and Rada Mihalcea. 2020 · 2020
Cited alongside, same era.
Tutorialvqa: Question answering dataset for tutorial videos
Anthony Colas, Seokhwan Kim, Franck Dernoncourt, Siddhesh Gupte, Daisy Zhe Wang, and Doo Soon Kim. 2020 · 2020
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. 2020 · 2020
Cited alongside, same era.
Data augmentation techniques for the video question answering task
Alex Falcon, Oswald Lanz, and Giuseppe Serra. 2020 · 2020
Cited alongside, same era.
Video2commonsense: Generating commonsense descriptions to enrich video captioning
Zhiyuan Fang, Tejas Gokhale, Pratyay Banerjee, et al. 2020 · 2020
Cited alongside, same era.
Knowledge-based video question answering with unsupervised scene descriptions
Noa Garcia and Yuta Nakashima. 2020 · 2020
Cited alongside, same era.
Knowit vqa: Answering knowledge-based questions about videos
Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. 2020 · 2020
Cited alongside, same era.

Then

Hierarchical conditional relation networks for multimodal video question answering
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. 2021 · 2021
Later among the works it cites.
Less is more: Clipbert for video-and-language learning via sparse sampling
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021 · 2021
Later among the works it cites.
Relation-aware hierarchical attention framework for video question answering
Fangtao Li, Ting Bai, Chenyu Cao, Zihe Liu, Chenghao Yan, and Bin Wu. 2021a · 2021
Later among the works it cites.
Bridge to answer: Structure-aware graph interaction network for video question answering
Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. 2021 · 2021
Later among the works it cites.
Recent advances in video question answering: A review of datasets and methods
Devshree Patel, Ratnam Parikh, and Yesha Shastri. 2021 · 2021
Later among the works it cites.
Progressive graph attention network for video question answering
Liang Peng, Shuangji Yang, Yi Bin, and Guoqing Wang. 2021 · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021 · 2021
Later among the works it cites.
Video question answering with phrases via semantic roles
Arka Sadhu, Kan Chen, and Ram Nevatia. 2021 · 2021
Later among the works it cites.
Video visual relation detection via iterative inference
Xindi Shang, Yicong Li, Junbin Xiao, Wei Ji, and Tat-Seng Chua. 2021 · 2021
Later among the works it cites.
Human-adversarial visual question answering
Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Magana, Tristan Thrush, Wojciech Galuba, Devi Parikh, and Douwe Kiela. 2021 · 2021
Later among the works it cites.
Dualvgr: A dual-visual graph reasoning unit for video question answering
Jianyu Wang, Bingkun Bao, and Changsheng Xu. 2021 · 2021
Later among the works it cites.
Transferring domain-agnostic knowledge in video question answering
Tianran Wu, Noa Garcia, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Haruo Takemura. 2021b · 2021
Later among the works it cites.
Next-qa: Next phase of question-answering to explaining temporal actions
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021 · 2021
Later among the works it cites.
Sutd-trafficqa: A question answering benchmark and an efficient network for video reasoning over traffic events
Li Xu, He Huang, and Jun Liu. 2021 · 2021
Later among the works it cites.
Learning from inside: Self-driven siamese sampling and reasoning for video question answering
Weijiang Yu, Haoteng Zheng, Mengfei Li, Lei Ji, Lijun Wu, Nong Xiao, and Nan Duan. 2021 · 2021
Later among the works it cites.
Merlot: Multimodal neural script knowledge models
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021 · 2021
Later among the works it cites.
Revisiting the" video" in video-language understanding
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. 2022 · 2022
Closest in time.
Vlp: A survey on vision-language pre-training
Original
Feilong Chen, Duzhen Zhang, Minglun Han, Xiuyi Chen, Jing Shi, Shuang Xu, and Bo Xu. 2022 · 2022
Closest in time.
(2.5+ 1) d spatio-temporal scene graphs for video question answering
Anoop Cherian, Chiori Hori, Tim K Marks, and Jonathan Le Roux. 2022 · 2022
Closest in time.
Matching structure for dual learning
Hao Fei, Shengqiong Wu, Yafeng Ren, and Meishan Zhang. 2022 · 2022
Closest in time.
Searching for computer vision north stars
Li Fei-Fei and Ranjay Krishna. 2022 · 2022
Closest in time.
Measuring compositional consistency for video question answering
Mona Gandhi, Mustafa Omer Gul, Eva Prakash, Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. 2022 · 2022
Closest in time.
Newskvqa: Knowledge-aware news video question answering
Original
Pranay Gupta and Manish Gupta. 2022 · 2022
Closest in time.
Vquad: Video question answering diagnostic dataset
Vivek Gupta, Badri N Patro, Hemant Parihar, and Vinay P Namboodiri. 2022 · 2022
Closest in time.
Deepstory: video story qa by deep embedded memory networks
Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. 2017 · 2022
Closest in time.
Dynamic multistep reasoning based on video scene graph for video question answering
Jianguo Mao, Wenbin Jiang, Xiangdong Wang, Zhifan Feng, Yajuan Lyu, Hong Liu, and Yong Zhu. 2022 · 2022
Closest in time.
Multilevel hierarchical network with multiscale sampling for video question answering
Min Peng, Chongyang Wang, Yuan Gao, Yu Shi, and Xiang-Dong Zhou. 2022 · 2022
Closest in time.
Video question answering with iterative video-text co-tokenization
AJ Piergiovanni, Kairo Morton, Weicheng Kuo, Michael S Ryoo, and Anelia Angelova. 2022 · 2022
Closest in time.
Dynamic spatio-temporal modular network for video question answering
Zi Qian, Xin Wang, Xuguang Duan, Hongyang Chen, and Wenwu Zhu. 2022 · 2022
Closest in time.
Fine-grained scene graph generation with data transfer
Ao Zhang, Yuan Yao, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. 2022 · 2022
Closest in time.
Pano-avqa: Grounded audio-visual question answering on 360deg videos
Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. 2021 · 2041
Closest in time.
Adversarial vqa: A new benchmark for evaluating the robustness of vqa models
Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. 2021b · 2051
Closest in time.
Hero: Hierarchical encoder for video+ language omni-representation pre-training
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020 · 2065
Closest in time.
On the hidden treasure of dialog in video question answering
Deniz Engin, François Schnitzler, Ngoc QK Duong, and Yannis Avrithis. 2021 · 2073
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…