Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning

Understand

Multimodal learning with incomplete modality is practical and challenging.

• Recently, researchers have focused on enhancing the robustness of pre-trained MultiModal Transformers (MMTs) under missing modality conditions by applying learnable prompts.
• However, these prompt-based methods face several limitations: (1) incomplete modalities provide restricted modal cues for task-specific inference, (2) dummy imputation for missing content causes information loss and introduces noise, and (3) static prompts are instance-agnostic, offering limited knowledge for instances with various missing conditions.
• To address these issues, we propose RAGPT, a novel Retrieval-AuGmented dynamic Prompt Tuning framework.