A survey on knowledge-enhanced multimodal learning

Understand

Multimodal learning has been a field of increasing interest, aiming to combine various modalities in a single joint representation.

• Especially in the area of visiolinguistic (VL) learning multiple models and techniques have been developed, targeting a variety of tasks that involve images and text.
• VL models have reached unprecedented performances by extending the idea of Transformers, so that both modalities can learn from each other.
• Massive pre-training procedures enable VL models to acquire a certain level of real-world understanding, although many gaps can be identified: the limited comprehension of commonsense, factual, temporal and other everyday knowledge aspects questions the extendability of VL tasks.