VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation

Understand

Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination.

• Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills.
• In this paper, we introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation.
• We improve prior learning by incorporating hand motion data, providing more effective guidance for dual-hand coordination than binary tactile feedback.