MSAF: Multimodal Split Attention Fusion

Understand

Multimodal learning mimics the reasoning process of the human multi-sensory system, which is used to perceive the surrounding world.

• While making a prediction, the human brain tends to relate crucial cues from multiple sources of information.
• In this work, we propose a novel multimodal fusion module that learns to emphasize more contributive features across all modalities.
• Specifically, the proposed Multimodal Split Attention Fusion (MSAF) module splits each modality into channel-wise equal feature blocks and creates a joint representation that is used to generate soft attention for each channel across the feature blocks.