Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding

Understand

Deep multimodal semantic understanding that goes beyond the mere superficial content relation mining has received increasing attention in the realm of artificial intelligence.

• The challenges of collecting and annotating high-quality multi-modal data have underscored the significance of few-shot learning.
• In this paper, we focus on two critical tasks under this context: few-shot multi-modal sarcasm detection (MSD) and multi-modal sentiment analysis (MSA).
• To address them, we propose Mixture-of-Prompt-Experts with Block-Aware Prompt Fusion (MoPE-BAF), a novel multi-modal soft prompt framework based on the unified vision-language model (VLM).