2018

Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

Sharma, Pratyusha, Mohan, Lekha, Pinto, Lerrel et al.

Understand

In recent years, we have seen an emergence of data-driven approaches in robotics.

However, most existing efforts and datasets are either in simulation or focus on a single task in isolation such as grasping, pushing or poking.
In order to make progress and capture the space of manipulation, we would need to collect a large-scale dataset of diverse tasks such as pouring, opening bottles, stacking objects etc.
But how does one collect such a dataset? In this paper, we present the largest available robotic-demonstration dataset (MIME) that contains 8260 human-robot demonstrations over 20 different robotic tasks (https://sites.google.com/view/mimedataset).

Built on

Eigenfaces for recognition
M. Turk and A. Pentland · 1991
Earlier work this paper cites.
The penn treebank: annotating predicate argument structure
M. Marcus, G. Kim, M. A. Marcinkiewicz, R. MacIntyre, A. Bies, M. Ferguson, K. Katz, and B. Schasberger · 1994
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Learning a sparse representation for object detection
S. Agarwal and D. Roth · 2002
Earlier work this paper cites.
Graspit! a versatile simulator for robotic grasping
A. T. Miller and P. K. Allen · 2004
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
A survey of robot learning from demonstration
B. D. Argall, S. Chernova, M. Veloso, and B. Browning · 2009
Earlier work this paper cites.
The pascal visual object classes (voc) challenge
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman · 2010
Earlier work this paper cites.
Autonomous helicopter aerobatics through apprenticeship learning
P. Abbeel, A. Coates, and A. Y. Ng · 2010
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
E. Todorov, T. Erez, and Y. Tassa · 2012
Earlier work this paper cites.
A benchmark for the evaluation of rgb-d slam systems
J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Vision meets robotics: The kitti dataset
A. Geiger, P. Lenz, C. Stiller, and R. Urtasun · 2013
Earlier work this paper cites.
Reinforcement learning in robotics: A survey
J. Kober, J. A. Bagnell, and J. Peters · 2013
Earlier work this paper cites.
Learning to select and generalize striking movements in robot table tennis
K. Mülling, J. Kober, O. Kroemer, and J. Peters · 2013
Earlier work this paper cites.
Learning monocular reactive uav control in cluttered natural environments
S. Ross, N. Melik-Barkhudarov, K. S. Shankar, A. Wendel, D. Dey, J. A. Bagnell, and M. Hebert · 2013
Earlier work this paper cites.

Similar

Two-stream convolutional networks for action recognition in videos
K. Simonyan and A. Zisserman · 2014
Cited alongside, same era.
Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling
Y. Gao, S. S. Vedula, C. E. Reiley, N. Ahmidi, B. Varadarajan, H. C. Lin, L. Tao, L. Zappella, B. Béjar, D. D. Yuh, et al · 2014
Cited alongside, same era.
Rich feature hierarchies for accurate object detection and semantic segmentation
R. Girshick, J. Donahue, T. Darrell, and J. Malik · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
D. P. Kingma and J. Ba · 2014
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Original
K. Simonyan and A. Zisserman · 2014
Cited alongside, same era.
Deep learning for detecting robotic grasps
I. Lenz, H. Lee, and A. Saxena · 2015
Cited alongside, same era.
Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours
L. Pinto and A. Gupta · 2016
Cited alongside, same era.
Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection
Original
S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen · 2016
Cited alongside, same era.
Learning to poke by poking: Experiential learning of intuitive physics
P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine · 2016
Cited alongside, same era.
Unsupervised learning for physical interaction through video prediction
C. Finn, I. Goodfellow, and S. Levine · 2016
Cited alongside, same era.
Openai gym
Original
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba · 2016
Cited alongside, same era.
The curious robot: Learning visual representations via physical interactions
L. Pinto, D. Gandhi, Y. Han, Y.-L. Park, and A. Gupta · 2016
Cited alongside, same era.

Then

Learning to push by grasping: Using multiple tasks for effective learning
L. Pinto and A. Gupta · 2017
Later among the works it cites.
Combining self-supervised learning and imitation for vision-based rope manipulation
A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine · 2017
Later among the works it cites.
Supervision via competition: Robot adversaries for learning tasks
L. Pinto, J. Davidson, and A. Gupta · 2017
Later among the works it cites.
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
S. Gu, E. Holly, T. Lillicrap, and S. Levine · 2017
Later among the works it cites.
Dex-net 3.0: Computing robust robot suction grasp targets in point clouds using a new analytic model and deep learning
Original
J. Mahler, M. Matl, X. Liu, A. Li, D. Gealy, and K. Goldberg · 2017
Later among the works it cites.
Carla: An open urban driving simulator
Original
A. Dosovitskiy, G. Ros, F. Codevilla, A. López, and V. Koltun · 2017
Later among the works it cites.
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi · 2017
Later among the works it cites.
Semantic scene completion from a single depth image
S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser · 2017
Later among the works it cites.
Matterport3d: Learning from rgb-d data in indoor environments
Original
A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Nießner, M. Savva, S. Song, A. Zeng, and Y. Zhang · 2017
Later among the works it cites.
Deep imitation learning for complex manipulation tasks from virtual reality teleoperation
Original
T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel · 2017
Later among the works it cites.
One-shot visual imitation learning via meta-learning
Original
C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine · 2017
Later among the works it cites.
Cassl: Curriculum accelerated self-supervised learning
A. Murali, L. Pinto, D. Gandhi, and A. Gupta · 2018
Closest in time.
Airsim: High-fidelity visual and physical simulation for autonomous vehicles
S. Shah, D. Dey, C. Lovett, and A. Kapoor · 2018
Closest in time.
Asymmetric actor critic for image-based robot learning
L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel · 2018
Closest in time.
Non-local neural networks
X. Wang, R. Girshick, A. Gupta, and K. He · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…