Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification

Understand

In this paper, we introduce a novel Synchronized Class Token Fusion (SCT Fusion) architecture in the framework of multi-modal multi-label classification (MLC) of remote sensing (RS) images.

• The proposed architecture leverages modality-specific attention-based transformer encoders to process varying input modalities, while exchanging information across modalities by synchronizing the special class tokens after each transformer encoder block.
• The synchronization involves fusing the class tokens with a trainable fusion transformation, resulting in a synchronized class token that contains information from all modalities.
• As the fusion transformation is trainable, it allows to reach an accurate representation of the shared features among different modalities.