Affordance-based Robot Manipulation with Flow Matching

Understand

We present a framework for assistive robot manipulation, which focuses on two fundamental challenges: first, efficiently adapting large-scale models to downstream scene affordance understanding tasks, especially in daily living scenarios where gathering multi-task data involving humans requires strenuous effort; second, effectively learning robot action trajectories by grounding the visual affordance model.

• We tackle the first challenge by employing a parameter-efficient prompt tuning method that prepends learnable text prompts to the frozen vision model to predict manipulation affordances in multi-task scenarios.
• Then we propose to learn robot action trajectories guided by affordances in a supervised flow matching method.
• Flow matching represents a robot visuomotor policy as a conditional process of flowing random waypoints to desired robot action trajectories.