Learning Factorized Multimodal Representations

Understand

Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information.

• Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing.
• In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels.
• We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors.