Improving Multimodal Accuracy Through Modality Pre-training and Attention

Understand

Training a multimodal network is challenging and it requires complex architectures to achieve reasonable performance.

• We show that one reason for this phenomena is the difference between the convergence rate of various modalities.
• We address this by pre-training modality-specific sub-networks in multimodal architectures independently before end-to-end training of the entire network.
• Furthermore, we show that the addition of an attention mechanism between sub-networks after pre-training helps identify the most important modality during ambiguous scenarios boosting the performance.