Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions

Understand

Multimodal deep learning systems which employ multiple modalities like text, image, audio, video, etc., are showing better performance in comparison with individual modalities (i.e., unimodal) systems.

• Multimodal machine learning involves multiple aspects: representation, translation, alignment, fusion, and co-learning.
• In the current state of multimodal machine learning, the assumptions are that all modalities are present, aligned, and noiseless during training and testing time.
• However, in real-world tasks, typically, it is observed that one or more modalities are missing, noisy, lacking annotated data, have unreliable labels, and are scarce in training or testing and or both.