OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulation

Understand

Vision-language-action (VLA) models have shown strong generalization for robotic action prediction through large-scale vision-language pretraining.

• However, most existing models rely solely on RGB cameras, limiting their perception and, consequently, manipulation capabilities.
• We present OmniVLA, an omni-modality VLA model that integrates novel sensing modalities for physically-grounded spatial intelligence beyond RGB perception.
• The core of our approach is the sensor-masked image, a unified representation that overlays spatially grounded and physically meaningful masks onto the RGB images, derived from sensors including an infrared camera, a mmWave radar, and a microphone array.