Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model

Understand

The essence of multi-modal fusion lies in exploiting the complementary information inherent in diverse modalities.

• However, prevalent fusion methods rely on traditional neural architectures and are inadequately equipped to capture the dynamics of interactions across modalities, particularly in presence of complex intra- and inter-modality correlations.
• Recent advancements in State Space Models (SSMs), notably exemplified by the Mamba model, have emerged as promising contenders.
• Particularly, its state evolving process implies stronger modality fusion paradigm, making multi-modal fusion on SSMs an appealing direction.