Discovering Spatio-Temporal Rationales for Video Question Answering

Understand

This paper strives to solve complex video question answering (VideoQA) which features long video containing multiple objects and events at different time.

• To tackle the challenge, we highlight the importance of identifying question-critical temporal moments and spatial objects from the vast amount of video content.
• Towards this, we propose a Spatio-Temporal Rationalization (STR), a differentiable selection module that adaptively collects question-critical moments and objects using cross-modal interaction.
• The discovered video moments and objects are then served as grounded rationales to support answer reasoning.