KnowIT VQA: Answering Knowledge-Based Questions about Videos

Understand

We propose a novel video understanding task by fusing knowledge-based and video question answering.

• First, we introduce KnowIT VQA, a video dataset with 24,282 human-generated question-answer pairs about a popular sitcom.
• The dataset combines visual, textual and temporal coherence reasoning together with knowledge-based questions, which need of the experience obtained from the viewing of the series to be answered.
• Second, we propose a video understanding model by combining the visual and textual video content with specific knowledge about the show.