Scaffolding Dexterous Manipulation with Vision-Language Models

Understand

Dexterous robotic hands are essential for performing complex manipulation tasks, yet remain difficult to train due to the challenges of demonstration collection and high-dimensional control.

• While reinforcement learning (RL) can alleviate the data bottleneck by generating experience in simulation, it typically relies on carefully designed, task-specific reward functions, which hinder scalability and generalization.
• Thus, contemporary works in dexterous manipulation have often bootstrapped from reference trajectories.
• These trajectories specify target hand poses that guide the exploration of RL policies and object poses that enable dense, task-agnostic rewards.