DSTC8-AVSD: Multimodal Semantic Transformer Network with Retrieval Style Word Generator

Understand

Audio Visual Scene-aware Dialog (AVSD) is the task of generating a response for a question with a given scene, video, audio, and the history of previous turns in the dialog.

• Existing systems for this task employ the transformers or recurrent neural network-based architecture with the encoder-decoder framework.
• Even though these techniques show superior performance for this task, they have significant limitations: the model easily overfits only to memorize the grammatical patterns; the model follows the prior distribution of the vocabularies in a dataset.
• To alleviate the problems, we propose a Multimodal Semantic Transformer Network.