Connecting Touch and Vision via Cross-Modal Prediction

Understand

Humans perceive the world using multi-modal sensory inputs such as vision, audition, and touch.

• In this work, we investigate the cross-modal connection between vision and touch.
• The main challenge in this cross-domain modeling task lies in the significant scale discrepancy between the two: while our eyes perceive an entire visual scene at once, humans can only feel a small region of an object at any given moment.
• To connect vision and touch, we introduce new tasks of synthesizing plausible tactile signals from visual inputs as well as imagining how we interact with objects given tactile data as input.