2022

From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation

Qin, Yuzhe, Su, Hao, Wang, Xiaolong

Understand

We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand.

We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer.
One key contribution of our system is that we construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand.
This provides an intuitive interface and avoid unstable human-robot hand retargeting for data collection, leading to large-scale and high quality data.

Built on

D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in
1989
Earlier work this paper cites.
I.-M. Chen and J. W. Burdick, “Finding antipodal point grasps on irregularly shaped objects,”
1993
Earlier work this paper cites.
D. Rus, “In-hand dexterous manipulation of piecewise-smooth 3-d objects,”
1999
Earlier work this paper cites.
A. M. Okamura, N. Smaby, and M. R. Cutkosky, “An overview of dexterous manipulation,” in
2000
Earlier work this paper cites.
A. Y. Ng, S. J. Russell,
2000
Earlier work this paper cites.
P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” 2004
2004
Earlier work this paper cites.
J. Kofman, S. Verma, and X. Wu, “Robot-manipulator teleoperation by markerless vision-based hand-arm tracking,”
2007
Earlier work this paper cites.
S. Kockara, T. Halic, K. Iqbal, C. Bayrak, and R. Rowe, “Collision detection: A survey,” in
2007
Earlier work this paper cites.
J. Peters and S. Schaal, “Reinforcement learning of motor skills with policy gradients,”
2008
Earlier work this paper cites.
G.-L. Du, P. Zhang, L.-Y. Yang, and Y.-B. Su, “Robot teleoperation using a vision-based manipulation method,” in
2010
Earlier work this paper cites.
Z. Zhang, “Microsoft kinect sensor and its effect,”
2012
Earlier work this paper cites.
G. Du, P. Zhang, J. Mai, and Z. Li, “Markerless kinect-based hand tracking for robot teleoperation,”
2012
Earlier work this paper cites.
V. Kumar, Z. Xu, and E. Todorov, “Fast, strong and compliant pneumatic actuation for dexterous tendon-driven hands,” in
2013
Earlier work this paper cites.
S. Andrews and P. G. Kry, “Goal directed multi-finger manipulation: Control policies and analysis,”
2013
Earlier work this paper cites.
I. Almetwally and M. Mallem, “Real-time tele-operation and tele-walking of humanoid robot nao using kinect depth camera,” in
2013
Earlier work this paper cites.
Y. Bai and C. K. Liu, “Dexterous manipulation using both palm and fingers,” 2014
2014
Earlier work this paper cites.
V. Kumar and E. Todorov, “Mujoco haptix: A virtual reality system for hand manipulation,” in
2015
Earlier work this paper cites.
B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols,”
2015
Earlier work this paper cites.

Similar

J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in
2015
Cited alongside, same era.
J. Ho and S. Ermon, “Generative adversarial imitation learning,” in
2016
Cited alongside, same era.
Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking deep reinforcement learning for continuous control,” 2016
2016
Cited alongside, same era.
J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,”
2017
Cited alongside, same era.
M. Večerík, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller, “Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards,”
2017
Cited alongside, same era.
R. M. Murray, Z. Li, and S. S. Sastry,
2017
Cited alongside, same era.
OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning dexterous in-hand manipulation,”
2018
Cited alongside, same era.
A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,” in
2018
Cited alongside, same era.
H. Hedayati, M. Walker, and D. Szafir, “Improving collocated robot teleoperation with augmented reality,” in
2018
Cited alongside, same era.
T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” in
2018
Cited alongside, same era.
D. Antotsiou, G. Garcia-Hernando, and T.-K. Kim, “Task-oriented hand motion retargeting for dexterous manipulation imitation,” in
2018
Cited alongside, same era.
F. Torabi, G. Warnell, and P. Stone, “Generative adversarial imitation from observation,”
2018
Cited alongside, same era.

Then

S. Li, X. Ma, H. Liang, M. Görner, P. Ruppel, B. Fang, F. Sun, and J. Zhang, “Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network,” in
2019
Later among the works it cites.
G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black, “Expressive body capture: 3d hands, face, and body from a single image,” in
2019
Later among the works it cites.
A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox, “Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system,” in
2020
Later among the works it cites.
S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto, “Visual imitation made easy,”
2020
Later among the works it cites.
K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn, “Reinforcement learning with videos: Combining offline observations with interaction,”
2020
Later among the works it cites.
L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg, “Concept2robot: Learning manipulation concepts from instructions and human demonstrations,” in
2020
Later among the works it cites.
S. Song, A. Zeng, J. Lee, and T. Funkhouser, “Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations,”
2020
Later among the works it cites.
G. Garcia-Hernando, E. Johns, and T.-K. Kim, “Physics-based dexterous manipulations with estimated hand poses and residual reinforcement learning,”
2020
Later among the works it cites.
M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki, “Graph-structured visual imitation,” in
2020
Later among the works it cites.
F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang,
2020
Later among the works it cites.
F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, “Mediapipe hands: On-device real-time hand tracking,”
Original
2020
Later among the works it cites.
Y. Rong, T. Shiratori, and H. Joo, “Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration,”
Original
2020
Later among the works it cites.
Y. Pan, C. Chen, D. Li, Z. Zhao, and J. Hong, “Augmented reality-based robot teleoperation system using rgb-d imaging and attitude teaching device,”
2021
Later among the works it cites.
I. Radosavovic, X. Wang, L. Pinto, and J. Malik, “State-only imitation learning for dexterous manipulation,”
2021
Later among the works it cites.
H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg, “Learning by watching: Physical imitation of manipulation skills from human videos,”
2021
Later among the works it cites.
Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang, “Dexmv: Imitation learning for dexterous manipulation from human videos,”
Original
2021
Later among the works it cites.
A. Sivakumar, K. Shaw, and D. Pathak, “Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube,”
Original
2022
Closest in time.
S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto, “Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation,”
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…