A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation

Understand

Multi-modal neural machine translation (NMT) aims to translate source sentences into a target language paired with images.

• However, dominant multi-modal NMT models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, which have potential to refine multi-modal representation learning.
• To deal with this issue, in this paper, we propose a novel graph-based multi-modal fusion encoder for NMT.
• Specifically, we first represent the input sentence and image using a unified multi-modal graph, which captures various semantic relationships between multi-modal semantic units (words and visual objects).