Multi-Layered Reasoning from a Single Viewpoint for Learning See-Through Grasping

Understand

Sensory substitution enables biological systems to perceive stimuli that are typically perceived by another organ, which is inspirational for physical agents.

• Multimodal perception of intrinsic and extrinsic interactions is critical in building an intelligent robot that learns.
• This study presents a Vision-based See-Through Perception (VBSeeThruP) architecture that simultaneously perceives multiple intrinsic and extrinsic modalities from a single visual input, in a markerless manner, all packed into a soft robotic finger using the Soft Polyhedral Network design.
• It is generally applicable to miniature vision systems placed beneath deformable networks with a see-through design, capturing real-time images of the network's physical interactions induced by contact-based events, overlaid on the visual scene of the external environment, as demonstrated in the ablation study.