From FiLM to Video: Multi-turn Question Answering with Multi-modal Context

Understand

Understanding audio-visual content and the ability to have an informative conversation about it have both been challenging areas for intelligent systems.

• The Audio Visual Scene-aware Dialog (AVSD) challenge, organized as a track of the Dialog System Technology Challenge 7 (DSTC7), proposes a combined task, where a system has to answer questions pertaining to a video given a dialogue with previous question-answer pairs and the video itself.
• We propose for this task a hierarchical encoder-decoder model which computes a multi-modal embedding of the dialogue context.
• It first embeds the dialogue history using two LSTMs.