2019

Simitate: A Hybrid Imitation Learning Benchmark

Memmesheimer, Raphael, Mykhalchyshyna, Ivanna, Seib, Viktor et al.

Understand

We present Simitate --- a hybrid benchmarking suite targeting the evaluation of approaches for imitation learning.

A dataset containing 1938 sequences where humans perform daily activities in a realistic environment is presented.
The dataset is strongly coupled with an integration into a simulator.
RGB and depth streams with a resolution of 960$\mathbb{\times}$540 at 30Hz and accurate ground truth poses for the demonstrator's hand, as well as the object in 6 DOF at 120Hz are provided.

Built on

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on pattern analysis and machine intelligence , vol. 22, 2000
2000
Earlier work this paper cites.
N. P. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator.” in IROS , vol. 4. Citeseer, 2004, pp. 2149–2154
2004
Earlier work this paper cites.
A. Alissandrakis, C. L. Nehaniv, K. Dautenhahn, and J. Saunders, “Evaluation of robot imitation attempts: comparison of the system’s and the human’s perspectives,” Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction , pp. 134–141, 2006. [Online]. Available: http://portal.acm.org/citation.cfm?id=1121265
2006
Earlier work this paper cites.
A. Alissandrakis, C. L. Nehaniv, and K. Dautenhahn, “Correspondence mapping induced state and action metrics for robotic imitation,” IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics , vol. 37, no. 2, pp. 299–307, 2007
2007
Earlier work this paper cites.
K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: the clear mot metrics,” Journal on Image and Video Processing , vol. 2008, p. 1, 2008
2008
Earlier work this paper cites.
B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and autonomous systems , vol. 57, no. 5, pp. 469–483, 2009
2009
Earlier work this paper cites.
T. Wisspeintner, T. Van Der Zant, L. Iocchi, and S. Schiffer, “Robocup@ home: Scientific competition and benchmarking for domestic service robots,” Interaction Studies , vol. 10, no. 3, pp. 392–426, 2009
2009
Earlier work this paper cites.
R. Kümmerle, B. Steder, C. Dornhege, M. Ruhnke, G. Grisetti, C. Stachniss, and A. Kleiner, “On measuring the accuracy of slam algorithms,” Autonomous Robots , vol. 27, no. 4, p. 387, 2009
2009
Earlier work this paper cites.
S. Ross, G. J. Gordon, and J. A. Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,” Proceedings of AISTATS , vol. 15, pp. 627–635, 2010. [Online]. Available: http://arxiv.org/abs/1011.0686
Original
2010
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems , 2012, pp. 1097–1105
2012
Earlier work this paper cites.
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on . IEEE, 2012, pp. 3354–3361
2012
Earlier work this paper cites.
J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of rgb-d slam systems,” in Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on . IEEE, 2012, pp. 573–580
2012
Earlier work this paper cites.
H. Pirsiavash and D. Ramanan, “Detecting activities of daily living in first-person camera views,” in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on . IEEE, 2012, pp. 2847–2854
2012
Earlier work this paper cites.
H. S. Koppula, R. Gupta, and A. Saxena, “Learning human activities and object affordances from rgb-d videos,” The International Journal of Robotics Research , vol. 32, no. 8, pp. 951–970, 2013
2013
Earlier work this paper cites.
Y. Wu, J. Lim, and M.-H. Yang, “Online object tracking: A benchmark,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2013, pp. 2411–2418
2013
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014 . Springer, 2014, pp. 740–755
2014
Earlier work this paper cites.
K. R. Amaro, T. Inamura, E. Dean-León, M. Beetz, and G. Cheng, “Bootstrapping humanoid robot skills by extracting semantic representations of human-like activities from virtual reality,” in IEEE-RAS International Conference on Humanoid Robots , 2014
2014
Earlier work this paper cites.

Similar

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 1–9
2015
Cited alongside, same era.
V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” arXiv preprint arXiv:1511.00561 , 2015
Original
2015
Cited alongside, same era.
P. O. Pinheiro, R. Collobert, and P. Dollár, “Learning to segment object candidates,” in NIPS , 2015
2015
Cited alongside, same era.
T. Gail, R. Hoffmann, M. Miezal, G. Bleser, and S. Leyendecker, “Towards bridging the gap between motion capturing and biomechanical optimal control simulations,” in Thematic Conference on Multibody Dynamics , 2015
2015
Cited alongside, same era.
S. Gupta and J. Malik, “Visual semantic role labeling,” arXiv preprint arXiv:1505.04474 , 2015
Original
2015
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Cited alongside, same era.
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 779–788
2016
Cited alongside, same era.
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in European conference on computer vision . Springer, 2016, pp. 21–37
2016
Cited alongside, same era.
S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional pose machines,” in CVPR , 2016
2016
Cited alongside, same era.
J. Ho and S. Ermon, “Generative adversarial imitation learning,” in Advances in Neural Information Processing Systems , 2016, pp. 4565–4573
2016
Cited alongside, same era.
P. Weinzaepfel, X. Martin, and C. Schmid, “Human action localization with sparse spatial supervision,” arXiv preprint arXiv:1605.05197 , 2016
Original
2016
Cited alongside, same era.
J. Zhang, W. Li, P. O. Ogunbona, P. Wang, and C. Tang, “Rgb-d-based action recognition datasets: A survey,” Pattern Recognition , vol. 60, pp. 86–105, 2016
2016
Cited alongside, same era.

Then

T. Simon, H. Joo, I. Matthews, and Y. Sheikh, “Hand keypoint detection in single images using multiview bootstrapping,” in CVPR , 2017
2017
Later among the works it cites.
Y. Duan, M. Andrychowicz, B. Stadie, O. J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,” in Advances in neural information processing systems , 2017, pp. 1087–1098
2017
Later among the works it cites.
M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg, “DART: Noise Injection for Robust Imitation Learning,” no. CoRL, pp. 1–14, 2017. [Online]. Available: http://arxiv.org/abs/1703.09327
Original
2017
Later among the works it cites.
Y. Duan, M. Andrychowicz, B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 1087–1098. [Online]. Available: http://papers.nips.cc/paper/6709-one-shot-imitation-learning
2017
Later among the works it cites.
T. Bates, K. Ramirez-Amaro, T. Inamura, and G. Cheng, “On-line simultaneous learning and recognition of everyday activities from virtual reality performances,” in Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on . IEEE, 2017, pp. 3510–3515
2017
Later among the works it cites.
Y. Mizuchi and T. Inamura, “Cloud-based multimodal human-robot interaction simulator utilizing ros and unity frameworks,” in System Integration (SII), 2017 IEEE/SICE International Symposium on . IEEE, 2017, pp. 948–955
2017
Later among the works it cites.
J. Leitner, A. W. Tow, N. Sünderhauf, J. E. Dean, J. W. Durham, M. Cooper, M. Eich, C. Lehnert, R. Mangels, C. McCool et al. , “The acrv picking benchmark: A robotic shelf picking benchmark to foster reproducible research,” in Robotics and Automation (ICRA), 2017 IEEE International Conference on . IEEE, 2017, pp. 4705–4712
2017
Later among the works it cites.
T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” arXiv preprint arXiv:1710.04615 , 2017
Original
2017
Later among the works it cites.
T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters et al. , “An algorithmic perspective on imitation learning,” Foundations and Trends® in Robotics , vol. 7, no. 1-2, pp. 1–179, 2018
2018
Later among the works it cites.
T. Gangwani and J. Peng, “Policy optimization by genetic distillation,” 2018
2018
Later among the works it cites.
T. Yu, C. Finn, S. Dasari, A. Xie, T. Zhang, P. Abbeel, and S. Levine, “One-shot imitation from observing humans via domain-adaptive meta-learning,” in Robotics: Science and Systems XIV, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018 , H. Kress-Gazit, S. Srinivasa, T. Howard, and N. Atanasov, Eds., 2018. [Online]. Available: http://www.roboticsproceedings.org/rss14/p02.html
2018
Later among the works it cites.
T. L. Paine, S. G. Colmenarejo, Z. Wang, S. E. Reed, Y. Aytar, T. Pfaff, M. W. Hoffman, G. Barth-Maron, S. Cabi, D. Budden, and N. de Freitas, “One-shot high-fidelity imitation: Training large-scale deep nets with RL,” CoRR , vol. abs/1810.05017, 2018. [Online]. Available: http://arxiv.org/abs/1810.05017
Original
2018
Later among the works it cites.
F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy, “End-to-end driving via conditional imitation learning,” in International Conference on Robotics and Automation (ICRA) , 2018
2018
Later among the works it cites.
V. Villani, B. Capelli, and L. Sabattini, “Use of Virtual Reality for the Evaluation of Human-Robot Interaction Systems in Complex Scenarios,” vol. Proceeding, pp. 422–427, 2018
2018
Later among the works it cites.
K.-K. Maninis, S. Caelles, J. Pont-Tuset, and L. Van Gool, “Deep extreme cut: From extreme points to object segmentation,” in Computer Vision and Pattern Recognition (CVPR) , 2018
2018
Later among the works it cites.
E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org , 2016–2019
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…