2022

RB2: Robotic Manipulation Benchmarking with a Twist

Dasari, Sudeep, Wang, Jianren, Hong, Joyce et al.

Understand

Benchmarks offer a scientific way to compare algorithms using objective performance metrics.

Good benchmarks have two features: (a) they should be widely useful for many research groups; (b) and they should produce reproducible findings.
In robotic manipulation research, there is a trade-off between reproducibility and broad accessibility.
If the benchmark is kept restrictive (fixed hardware, objects), the numbers are reproducible but the setup becomes less general.

Built on

The analysis of permutations
R. L. Plackett · 1975
Earlier work this paper cites.
The southampton hand: an intelligent myoelectric prosthesis
P. J. Kyberd and P. H. Chappell · 1994
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Robocup: The robot world cup initiative
H. Kitano, M. Asada, Y. Kuniyoshi, I. Noda, and E. Osawa · 1997
Earlier work this paper cites.
A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems
E. Todorov and W. Li · 2005
Earlier work this paper cites.
Dynamic movement primitives-a framework for motor control in humans and humanoid robotics
S. Schaal · 2006
Earlier work this paper cites.
robosuite: A modular simulation framework and benchmark for robot learning
Original
Y. Zhu, J. Wong, A. Mandlekar, and R. Martín-Martín · 2009
Earlier work this paper cites.
Relative entropy policy search
J. Peters, K. Mülling, and Y. Altün · 2010
Earlier work this paper cites.
Technology readiness levels for randomized bin picking
J. Marvel, R. Eastman, G. Cheok, K. Saidi, Tsai, and E. Messina · 2012
Earlier work this paper cites.
MuJoCo: A physics engine for model-based control
E. Todorov, T. Erez, and Y. Tassa · 2012
Earlier work this paper cites.
Aging, hypertension and physiological tremor: The contribution of the cardioballistic impulse to tremorgenesis in older adults
S. Morrison, J. J. Sosnoff, K. S. Heffernan, S. Y. Jae, and B. Fernhall · 2013
Earlier work this paper cites.
Dynamic movement primitives for human-robot interaction: Comparison with human behavioral observation
M. Prada, A. Remazeilles, A. Koene, and S. Endo · 2013
Earlier work this paper cites.
Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols
Original
B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar · 2015
Earlier work this paper cites.
Openai gym
Original
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba · 2016
Earlier work this paper cites.
Analysis and observations from the first amazon picking challenge
N. Correll, K. E. Bekris, D. Berenson, O. Brock, A. Causo, K. Hauser, K. Okada, A. Rodriguez, J. M. Romano, and P. R. Wurman · 2016
Earlier work this paper cites.
Hierarchical relative entropy policy search
C. Daniel, G. Neumann, O. Kroemer, and J. Peters · 2016
Earlier work this paper cites.

Similar

Deep spatial autoencoders for visuomotor learning
C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel · 2016
Cited alongside, same era.
Hindsight experience replay
M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba · 2017
Cited alongside, same era.
Performance metrics and test methods for robotic hands
J. Falco, K. Van Wyk, and E. Messina · 2018
Cited alongside, same era.
Deep reinforcement learning that matters
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger · 2018
Cited alongside, same era.
From virtual demonstration to real-world manipulation using lstm and mdn
R. Rahmatizadeh, P. Abolghasemi, A. Behal, and L. Bölöni · 2018
Cited alongside, same era.
ROBEL: RObotics BEnchmarks for Learning with low-cost robots
M. Ahn, H. Zhu, K. Hartikainen, H. Ponte, A. Gupta, S. Levine, and V. Kumar · 2019
Cited alongside, same era.
Benchmarking protocol for grasp planning algorithms
Y. Bekiroglu, N. Marturi, M. A. Roa, K. J. M. Adjigble, T. Pardi, C. Grimm, R. Balasubramanian, K. Hang, and R. Stolkin · 2019
Cited alongside, same era.
Benchmarking cluttered robot pick-and-place manipulation with the box and blocks test
A. S. Morgan, K. Hang, W. G. Bircher, F. M. Alladkani, A. Gandhi, B. Calli, and A. M. Dollar · 2019
Cited alongside, same era.
Pyrobot: An open-source robotics framework for research and benchmarking
Original
A. Murali, T. Chen, K. V. Alwala, D. Gandhi, L. Pinto, S. Gupta, and A. Gupta · 2019
Cited alongside, same era.
Replab: A reproducible low-cost arm benchmark platform for robotic learning
Original
B. Yang, J. Zhang, V. Pong, S. Levine, and D. Jayaraman · 2019
Cited alongside, same era.
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning
Original
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine · 2019
Cited alongside, same era.
Neural dynamic policies for end-to-end sensorimotor learning
S. Bahl, M. Mukadam, A. Gupta, and D. Pathak · 2020
Cited alongside, same era.

Then

D4rl: Datasets for deep data-driven reinforcement learning
Original
J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine · 2020
Later among the works it cites.
Benchmarking bimanual cloth manipulation
I. Garcia-Camacho, M. Lippi, M. C. Welle, H. Yin, R. Antonova, A. Varava, J. Borras, C. Torras, A. Marino, G. Alenya, et al · 2020
Later among the works it cites.
Rlbench: The robot learning benchmark & learning environment
S. James, Z. Ma, D. R. Arrojo, and A. J. Davison · 2020
Later among the works it cites.
Morel: Model-based offline reinforcement learning
Original
R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims · 2020
Later among the works it cites.
Conservative q-learning for offline reinforcement learning
Original
A. Kumar, A. Zhou, G. Tucker, and S. Levine · 2020
Later among the works it cites.
Benchmarking hand and grasp resilience to dynamic loads
F. Negrello, W. Friedl, G. Grioli, M. Garabini, O. Brock, A. Bicchi, M. A. Roa, and M. G. Catalano · 2020
Later among the works it cites.
Benchmarks for aerial manipulation
A. Suarez, V. M. Vega, M. Fernandez, G. Heredia, and A. Ollero · 2020
Later among the works it cites.
dm_control: Software and tasks for continuous control, 2020
Y. Tassa, S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, and N. Heess · 2020
Later among the works it cites.
Sapien: A simulated part-based interactive environment
F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al · 2020
Later among the works it cites.
Hierarchical neural dynamic policies
S. Bahl, A. Gupta, and D. Pathak · 2021
Later among the works it cites.
Pybullet, a python module for physics simulation for games, robotics and machine learning
E. Coumans and Y. Bai · 2021
Later among the works it cites.
Manipulathor: A framework for visual object manipulation
K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mottaghi · 2021
Later among the works it cites.
Benchmarking structured policies and policy optimization for real-world dexterous object manipulation
Original
N. Funk, C. Schaff, R. Madan, T. Yoneda, J. U. De Jesus, J. Watson, E. K. Gordon, F. Widmaier, S. Bauer, S. S. Srinivasa, et al · 2021
Later among the works it cites.
IKEA furniture assembly environment for long-horizon complex manipulation tasks
Y. Lee, E. S. Hu, and J. J. Lim · 2021
Later among the works it cites.
What matters in learning from offline human demonstrations for robot manipulation
Original
A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…