Multimodal Dialogue State Tracking By QA Approach with Data Augmentation

Understand

Recently, a more challenging state tracking task, Audio-Video Scene-Aware Dialogue (AVSD), is catching an increasing amount of attention among researchers.

• Different from purely text-based dialogue state tracking, the dialogue in AVSD contains a sequence of question-answer pairs about a video and the final answer to the given question requires additional understanding of the video.
• This paper interprets the AVSD task from an open-domain Question Answering (QA) point of view and proposes a multimodal open-domain QA system to deal with the problem.
• The proposed QA system uses common encoder-decoder framework with multimodal fusion and attention.