RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing

Understand

Recent advances in self-supervised learning for Vision Transformers (ViTs) have fueled breakthroughs in remote sensing (RS) foundation models.

• However, the quadratic complexity of self-attention poses a significant barrier to scalability, particularly for large models and high-resolution images.
• While the linear-complexity Mamba architecture offers a promising alternative, existing RS applications of Mamba remain limited to supervised tasks on small, domain-specific datasets.
• To address these challenges, we propose RoMA, a framework that enables scalable self-supervised pretraining of Mamba-based RS foundation models using large-scale, diverse, unlabeled data.