Fetching the paper…

Reading the bibliography…

2023

MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation

Lancaster, Patrick, Hansen, Nicklas, Rajeswaran, Aravind et al.

Original
PDF
Post on X

Understand

Robotic systems that aspire to operate in uninstrumented real-world environments must perceive the world directly via onboard sensing.

Vision-based learning systems aim to eliminate the need for environment instrumentation by building an implicit understanding of the world based on raw pixels, but navigating the contact-rich high-dimensional search space from solely sparse visual reward signals significantly exacerbates the challenge of exploration.
The applicability of such systems is thus typically restricted to simulated or heavily engineered environments since agent exploration in the real-world without the guidance of explicit state estimation and dense rewards can lead to unsafe behavior and safety faults that are catastrophic.
In this study, we isolate the root causes behind these limitations to develop a system, called MoDem-V2, capable of learning contact-rich manipulation directly in the uninstrumented real world.

MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation · Around

Built on

C. G. Atkeson and S. Schaal, “Robot learning from demonstration,” in ICML , 1997
1997
Earlier work this paper cites.
L. Pinto and A. K. Gupta, “Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,” 2016 IEEE International Conference on Robotics and Automation (ICRA) , pp. 3406–3413, 2015
2015
Earlier work this paper cites.
A. A. Rusu, M. Vecerík, T. Rothörl, N. M. O. Heess, R. Pascanu, and R. Hadsell, “Sim-to-real robot learning from pixels with progressive nets,” ArXiv , vol. abs/1610.04286, 2016
Original
2016
Earlier work this paper cites.
S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” The International Journal of Robotics Research , vol. 37, pp. 421 – 436, 2016
2016
Earlier work this paper cites.
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 23–30, 2017
2017
Earlier work this paper cites.
Z. Huang, S. Zhou, B. Zhuang, and X. Zhou, “Learning to run with actor-critic ensemble,” arXiv preprint arXiv:1712.08987 , 2017
Original
2017
Earlier work this paper cites.
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) . IEEE, 2017, pp. 23–30
2017
Earlier work this paper cites.
D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” in Advances in Neural Information Processing Systems 31 . Curran Associates, Inc., 2018, pp. 2451–2463
2018
Earlier work this paper cites.
S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in International conference on machine learning . PMLR, 2018, pp. 1587–1596
2018
Earlier work this paper cites.
K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” Advances in neural information processing systems , vol. 31, 2018
2018
Earlier work this paper cites.
A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations,” in Proceedings of Robotics: Science and Systems (RSS) , 2018
2018
Earlier work this paper cites.
F. Ebert, C. Finn, S. Dasari, A. Xie, A. X. Lee, and S. Levine, “Visual foresight: Model-based deep reinforcement learning for vision-based robotic control,” ArXiv , vol. abs/1812.00568, 2018
Original
2018
Earlier work this paper cites.
Y. Chow, O. Nachum, E. Duenez-Guzman, and M. Ghavamzadeh, “A lyapunov-based approach to safe reinforcement learning,” Advances in neural information processing systems , vol. 31, 2018
2018
Earlier work this paper cites.
A. Wachi, Y. Sui, Y. Yue, and M. Ono, “Safe exploration and optimization of constrained mdps using gaussian processes,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 32, no. 1, 2018
2018
Earlier work this paper cites.
H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar, “Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost,” 2019 International Conference on Robotics and Automation (ICRA) , pp. 3651–3657, 2018
2018
Earlier work this paper cites.
D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, “Learning latent dynamics for planning from pixels,” in International Conference on Machine Learning , 2019, pp. 2555–2565
2019
Earlier work this paper cites.

Similar

Y. Ge, F. Zhu, X. Ling, and Q. Liu, “Safe q-learning method based on constrained markov decision processes,” IEEE Access , vol. 7, pp. 165 007–165 017, 2019
2019
Cited alongside, same era.
M. Henaff, A. Canziani, and Y. LeCun, “Model-predictive policy learning with uncertainty regularization for driving in dense traffic,” arXiv preprint arXiv:1901.02705 , 2019
Original
2019
Cited alongside, same era.
J.-B. Grill, F. Strub, F. Altch’e, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Á. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent: A new approach to self-supervised learning,” ArXiv , vol. abs/2006.07733, 2020
Original
2020
Cited alongside, same era.
“Robohive – a unified framework for robot learning,” https://sites.google.com/view/robohive , 2020. [Online]. Available: https://sites.google.com/view/robohive
2020
Cited alongside, same era.

Then

W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao, “Mastering atari games with limited data,” in NeurIPS , 2021
2021
Later among the works it cites.
D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman, “Mt-opt: Continuous multi-task robotic reinforcement learning at scale,” ArXiv , vol. abs/2104.08212, 2021
Original
2021
Later among the works it cites.
A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. V. Wyk, A. Zhurkevich, B. Sundaralingam, Y. S. Narang, J.-F. Lafleche, D. Fox, and G. State, “Dextreme: Transfer of agile in-hand manipulation from simulation to reality,” ArXiv , vol. abs/2210.13702, 2022
Original
2022
Later among the works it cites.
N. Hansen, X. Wang, and H. Su, “Temporal difference learning for model predictive control,” arXiv preprint arXiv:2203.04955 , 2022
Original
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin, “A framework for efficient robotic manipulation,” ArXiv , vol. abs/2012.07975, 2020

2020

Cited alongside, same era.

M. Ahn, H. Zhu, K. Hartikainen, H. Ponte, A. Gupta, S. Levine, and V. Kumar, “Robel: Robotics benchmarks for learning with low-cost robots,” in Conference on robot learning . PMLR, 2020, pp. 1300–1313

2020

Cited alongside, same era.

L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, R. Sepassi, G. Tucker, and H. Michalewski, “Model-based reinforcement learning for atari,” ArXiv , vol. abs/1903.00374, 2020

2020

Cited alongside, same era.

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. P. Lillicrap, and D. Silver, “Mastering atari, go, chess and shogi by planning with a learned model,” Nature , vol. 588 7839, pp. 604–609, 2020

2020

Cited alongside, same era.

G. D. Kontes, D. D. Scherer, T. Nisslbeck, J. Fischer, and C. Mutschler, “High-speed collision avoidance using deep reinforcement learning and domain randomization for autonomous vehicles,” in 2020 IEEE 23rd international conference on Intelligent Transportation Systems (ITSC) . IEEE, 2020, pp. 1–8

2020

Cited alongside, same era.

B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull, “Active domain randomization,” in Conference on Robot Learning . PMLR, 2020, pp. 1162–1176

2020

Cited alongside, same era.

J. Zhang, B. Cheung, C. Finn, S. Levine, and D. Jayaraman, “Cautious adaptation for reinforcement learning in safety-critical settings,” in International Conference on Machine Learning . PMLR, 2020, pp. 11 055–11 065

2020

Cited alongside, same era.

B. Thananjeyan, A. Balakrishna, U. Rosolia, F. Li, R. McAllister, J. E. Gonzalez, S. Levine, F. Borrelli, and K. Goldberg, “Safety augmented value estimation from demonstrations (saved): Safe deep model-based rl for sparse cost robotic tasks,” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3612–3619, 2020

2020

Cited alongside, same era.

N. Hansen, Y. Lin, H. Su, X. Wang, V. Kumar, and A. Rajeswaran, “Modem: Accelerating visual model-based reinforcement learning with demonstrations,” arXiv preprint arXiv:2212.05698 , 2022

2022

Later among the works it cites.

R. Bhirangi, A. DeFranco, J. Adkins, C. Majidi, A. Gupta, T. Hellebrekers, and V. Kumar, “All the feels: A dexterous hand with large area sensing,” arXiv preprint arXiv:2210.15658 , 2022

2022

Later among the works it cites.

L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig, “Safe learning in robotics: From learning-based control to safe reinforcement learning,” Annual Review of Control, Robotics, and Autonomous Systems , vol. 5, pp. 411–444, 2022

2022

Later among the works it cites.

B. Yang, G. Habibi, P. Lancaster, B. Boots, and J. Smith, “Motivating physical activity via competitive human-robot interaction,” in Conference on Robot Learning . PMLR, 2022, pp. 839–849

2022

Later among the works it cites.

E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” in Conference on Robot Learning , 2022

2022

Later among the works it cites.

S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,” in Conference on Robot Learning , 2022

2022

Later among the works it cites.

Y. Zhu, A. Joshi, P. Stone, and Y. Zhu, “Viola: Imitation learning for vision-based manipulation with object proposal priors,” ArXiv , vol. abs/2210.11339, 2022

2022

Later among the works it cites.

A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. C. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. S. Ryoo, G. Salazar, P. R. Sanketi, K. Sayed, J. Singh, S. A. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. H. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich, “Rt-1: Robotics transformer for real-world control at scale,” ArXiv , vol. abs/2212.06817, 2022

2022

Later among the works it cites.

A. Kumar, A. Singh, F. Ebert, Y. Yang, C. Finn, and S. Levine, “Pre-training for robots: Offline rl enables learning new tasks from a handful of trials,” ArXiv , vol. abs/2210.05178, 2022

2022

Later among the works it cites.

B. Yang, L. Zheng, L. J. Ratliff, B. Boots, and J. R. Smith, “Stackelberg games for learning emergent behaviors during competitive autocurricula,” arXiv preprint arXiv:2305.03735 , 2023

2023

Closest in time.

P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg, “Daydreamer: World models for physical robot learning,” in Conference on Robot Learning . PMLR, 2023, pp. 2226–2240

2023

Closest in time.