CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders

Understand

A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes self-supervised learning algorithms invaluable.

• We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multimodal representations.
• Our method separately encodes masked-out multispectral optical and synthetic aperture radar samples -- aligned in space and time -- and performs cross-modal contrastive learning.
• Another encoder fuses these sensors, producing joint multimodal encodings that are used to predict the masked patches via a lightweight decoder.