M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation

Understand

One of the most critical aspects of multimodal Reinforcement Learning (RL) is the effective integration of different observation modalities.

• Having robust and accurate representations derived from these modalities is key to enhancing the robustness and sample efficiency of RL algorithms.
• However, learning representations in RL settings for visuotactile data poses significant challenges, particularly due to the high dimensionality of the data and the complexity involved in correlating visual and tactile inputs with the dynamic environment and task objectives.
• To address these challenges, we propose Multimodal Contrastive Unsupervised Reinforcement Learning (M2CURL).