Fetching the paper…

Reading the bibliography…

Gen2Sim: Scaling up Robot Learning in Simulation with Generative Models · Around

2023

Gen2Sim: Scaling up Robot Learning in Simulation with Generative Models

Katara, Pushkal, Xian, Zhou, Fragkiadaki, Katerina

Original
PDF
Post on X

Understand

Generalist robot manipulators need to learn a wide variety of manipulation skills across diverse environments.

Current robot training pipelines rely on humans to provide kinesthetic demonstrations or to program simulation environments and to code up reward functions for reinforcement learning.
Such human involvement is an important bottleneck towards scaling up robot learning across diverse tasks and environments.
We propose Generation to Simulation (Gen2Sim), a method for scaling up robot skill learning in simulation by automating generation of 3D assets, task descriptions, task decompositions and reward functions using large pre-trained generative models of language and vision.

Built on

D. Xu, R. Martín-Martín, D. Huang, Y. Zhu, S. Savarese, and L. Fei-Fei, “Regression planning networks,” CoRR , vol. abs/1909.13072, 2019. [Online]. Available: http://arxiv.org/abs/1909.13072
Original
1909
Earlier work this paper cites.
T. Migimatsu and J. Bohg, “Object-centric task and motion planning in dynamic environments,” CoRR , vol. abs/1911.04679, 2019. [Online]. Available: http://arxiv.org/abs/1911.04679
Original
1911
Earlier work this paper cites.
M. Toussaint, “Logic-geometric programming: An optimization-based approach to combined task and motion planning,” in Proceedings of the 24th International Conference on Artificial Intelligence , ser. IJCAI’15. AAAI Press, 2015, p. 1930–1936
1936
Earlier work this paper cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” 2020. [Online]. Available: https://arxiv.org/abs/2005.14165
Original
2005
Earlier work this paper cites.
P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, and G. Ranzuglia, “MeshLab: an Open-Source Mesh Processing Tool,” in Eurographics Italian Chapter Conference , V. Scarano, R. D. Chiara, and U. Erra, Eds. The Eurographics Association, 2008
2008
Earlier work this paper cites.
J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” CoRR , vol. abs/2010.11251, 2020. [Online]. Available: https://arxiv.org/abs/2010.11251
Original
2010
Earlier work this paper cites.
L. P. Kaelbling and T. Lozano-Pérez, “Hierarchical task and motion planning in the now,” in 2011 IEEE International Conference on Robotics and Automation , 2011, pp. 1470–1477
2011
Earlier work this paper cites.
Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu, “Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs,” CoRR , vol. abs/2012.07277, 2020. [Online]. Available: https://arxiv.org/abs/2012.07277
Original
2012
Earlier work this paper cites.
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ international conference on intelligent robots and systems . IEEE, 2012, pp. 5026–5033
2012
Earlier work this paper cites.
M. Macklin, M. Müller, N. Chentanez, and T.-Y. Kim, “Unified particle physics for real-time applications,” ACM Transactions on Graphics (TOG) , vol. 33, no. 4, pp. 1–12, 2014
2014
Earlier work this paper cites.
S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” 2016
2016
Earlier work this paper cites.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540 , 2016
Original
2016
Earlier work this paper cites.
E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org , 2016
2016
Earlier work this paper cites.
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” 2017
2017
Earlier work this paper cites.
E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, “Ai2-thor: An interactive 3d environment for visual ai,” arXiv preprint arXiv:1712.05474 , 2017
Original
2017
Earlier work this paper cites.
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” 2017
2017
Earlier work this paper cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017
2017
Earlier work this paper cites.
D. Lyu, F. Yang, B. Liu, and S. Gustafson, “SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning,” CoRR , vol. abs/1811.00090, 2018. [Online]. Available: http://arxiv.org/abs/1811.00090
Original
2018
Earlier work this paper cites.
C. R. Garrett, T. Lozano-Pérez, and L. P. Kaelbling, “Stripstream: Integrating symbolic planners and blackbox samplers,” CoRR , vol. abs/1802.08705, 2018. [Online]. Available: http://arxiv.org/abs/1802.08705
Original
2018
Earlier work this paper cites.
OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang, “Solving rubik’s cube with a robot hand,” 2019
2019
Earlier work this paper cites.
M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, “Habitat: A platform for embodied ai research,” 2019
2019
Earlier work this paper cites.
T. Nguyen-Phuoc, C. Li, L. Theis, C. Richardt, and Y.-L. Yang, “Hologan: Unsupervised learning of 3d representations from natural images,” 2019
2019
Earlier work this paper cites.
M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki, “Graph-structured visual imitation,” in Conference on Robot Learning . PMLR, 2020, pp. 979–989
2020
Earlier work this paper cites.
F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su, “Sapien: A simulated part-based interactive environment,” 2020
2020
Earlier work this paper cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” 2020
2020
Earlier work this paper cites.
H.-Y. F. Tung, Z. Xian, M. Prabhudesai, S. Lal, and K. Fragkiadaki, “3d-oes: Viewpoint-invariant object-factorized environment simulators,” arXiv preprint arXiv:2011.06464 , 2020
Original
2020
Earlier work this paper cites.
C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano et al. , “Threedworld: A platform for interactive multi-modal physical simulation,” arXiv preprint arXiv:2007.04954 , 2020
Original
2020
Earlier work this paper cites.
X. Lin, Y. Wang, J. Olkin, and D. Held, “Softgym: Benchmarking deep reinforcement learning for deformable object manipulation,” arXiv preprint arXiv:2011.07215 , 2020
Original
2020
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” CoRR , vol. abs/2103.00020, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020
Original
2021
Earlier work this paper cites.

Similar

S. Höfer, K. E. Bekris, A. Handa, J. C. Gamboa, M. Mozifian, F. Golemo, C. G. Atkeson, D. Fox, K. Goldberg, J. Leonard, C. K. Liu, J. Peters, S. Song, P. Welinder, and M. White, “Sim2real in robotics and automation: Applications and challenges,” IEEE Trans Autom. Sci. Eng. , vol. 18, no. 2, pp. 398–400, 2021. [Online]. Available: https://doi.org/10.1109/TASE.2021.3064065
2021
Cited alongside, same era.
A. Kumar, Z. Fu, D. Pathak, and J. Malik, “Rma: Rapid motor adaptation for legged robots,” 2021
2021
Cited alongside, same era.
S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K. Vainio, Z. Lian, C. Gokmen, S. Buch, C. K. Liu, S. Savarese, H. Gweon, J. Wu, and L. Fei-Fei, “Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments,” 2021
2021
Cited alongside, same era.

Then

Y. Ji, G. B. Margolis, and P. Agrawal, “Dribblebot: Dynamic legged manipulation in the wild,” 2023
2023
Closest in time.
E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Mueller, V. Koltun, and D. Scaramuzza, “Champion-level drone racing using deep reinforcement learning,” Nature , vol. 620, pp. 982–987, 08 2023
2023
Closest in time.
Z. Xian, T. Gervet, Z. Xu, Y.-L. Qiao, and T.-H. Wang, “Towards a foundation model for generalist robots: Diverse skill learning at scale via automated task and scene generation,” arXiv preprint arXiv:2305.10455 , 2023
Original
2023
Closest in time.
X. Zhu, Y. Chen, H. Tian, C. Tao, W. Su, C. Yang, G. Huang, B. Li, L. Lu, X. Wang, Y. Qiao, Z. Zhang, and J. Dai, “Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory,” 2023
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

C. Gan, J. Schwartz, S. Alter, D. Mrowca, M. Schrimpf, J. Traer, J. D. Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, K. Kim, E. Wang, M. Lingelbach, A. Curtis, K. Feigelis, D. M. Bear, D. Gutfreund, D. Cox, A. Torralba, J. J. DiCarlo, J. B. Tenenbaum, J. H. McDermott, and D. L. K. Yamins, “Threedworld: A platform for interactive multi-modal physical simulation,” 2021

2021

Cited alongside, same era.

P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” CoRR , vol. abs/2107.13586, 2021. [Online]. Available: https://arxiv.org/abs/2107.13586

2021

Cited alongside, same era.

E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein, “pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 5799–5809

2021

Cited alongside, same era.

M. J. McDonald and D. Hadfield-Menell, “Guided imitation of task and motion planning,” 2021

2021

Cited alongside, same era.

Z. Xian, S. Lal, H.-Y. Tung, E. A. Platanios, and K. Fragkiadaki, “Hyperdynamics: Meta-learning object and agent dynamics with hypernetworks,” arXiv preprint arXiv:2103.09439 , 2021

2021

Cited alongside, same era.

A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer et al. , “Perceiver io: A general architecture for structured inputs & outputs,” arXiv preprint arXiv:2107.14795 , 2021

2021

Cited alongside, same era.

V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State, “Isaac gym: High performance gpu-based physics simulation for robot learning,” 2021

2021

Cited alongside, same era.

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022

2022

Cited alongside, same era.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 684–10 695

2022

Cited alongside, same era.

S. Lifshitz, K. Paster, H. Chan, J. Ba, and S. McIlraith, “Steve-1: A generative model for text-to-behavior in minecraft,” 2023

2023

Closest in time.

G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, “Voyager: An open-ended embodied agent with large language models,” 2023

2023

Closest in time.

H. Ha, P. Florence, and S. Song, “Scaling up and distilling down: Language-guided robot skill acquisition,” arXiv preprint arXiv:2307.14535 , 2023

2023

Closest in time.

W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, “Voxposer: Composable 3d value maps for robotic manipulation with language models,” arXiv preprint arXiv:2307.05973 , 2023

2023

Closest in time.

W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik et al. , “Language to rewards for robotic skill synthesis,” arXiv preprint arXiv:2306.08647 , 2023

2023

Closest in time.

R. Chen, Y. Chen, N. Jiao, and K. Jia, “Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation,” arXiv preprint arXiv:2303.13873 , 2023

2023

Closest in time.

C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin, “Magic3d: High-resolution text-to-3d content creation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023

2023

Closest in time.

L. Melas-Kyriazi, C. Rupprecht, I. Laina, and A. Vedaldi, “Realfusion: 360 reconstruction of any object from a single image,” in CVPR , 2023. [Online]. Available: https://arxiv.org/abs/2302.10663

2023

Closest in time.

J. Tang, T. Wang, B. Zhang, T. Zhang, R. Yi, L. Ma, and D. Chen, “Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior,” 2023

2023

Closest in time.

B. Shen, X. Yan, C. R. Qi, M. Najibi, B. Deng, L. Guibas, Y. Zhou, and D. Anguelov, “Gina-3d: Learning to generate implicit neural assets in the wild,” 2023

2023

Closest in time.

K. Wang, W. R. J. I. au2, S. Lu, X. Huang, J. Booth, R. Kramer-Bottiglio, M. Aanjaneya, and K. Bekris, “Real2sim2real transfer for control of cable-driven robots via a differentiable physics engine,” 2023

2023

Closest in time.

M. Dalal, A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and D. Fox, “Imitating task and motion planning with visuomotor transformers,” 2023

2023

Closest in time.

A. Murali, A. Mousavian, C. Eppner, A. Fishman, and D. Fox, “Cabinet: Scaling neural collision detection for object rearrangement with procedural scene generation,” 2023

2023

Closest in time.

N. Gkanatsios, A. Jain, Z. Xian, Y. Zhang, C. Atkeson, and K. Fragkiadaki, “Energy-based models as zero-shot planners for compositional scene rearrangement,” arXiv preprint arXiv:2304.14391 , 2023

2023

Closest in time.

T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki, “Act3d: Infinite resolution action detection transformer for robotic manipulation,” arXiv preprint arXiv:2306.17817 , 2023

2023

Closest in time.

Z. Xian, N. Gkanatsios, T. Gervet, T.-w. Ke, and K. Fragkiadaki, “Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation,” Conference on Robot Learning , 2023

2023

Closest in time.

Z. Xian, B. Zhu, Z. Xu, H.-Y. Tung, A. Torralba, K. Fragkiadaki, and C. Gan, “Fluidlab: A differentiable environment for benchmarking complex fluid manipulation,” in International Conference on Learning Representations , 2023

2023

Closest in time.

T.-H. Wang, A. E. Spielberg, P. Ma, Z. Xian, H. Zhang, J. B. Tenenbaum, and C. Gan, “Softzoo: A soft robot co-design benchmark for locomotion in diverse environments,” in International Conference on Learning Representations , 2023

2023

Closest in time.

OpenAI, “Gpt-4 technical report,” 2023

2023

Closest in time.

Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee, “Gligen: Open-set grounded text-to-image generation,” arXiv preprint arXiv:2301.07093 , 2023

2023

Closest in time.

L. Zhang and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” arXiv preprint arXiv:2302.05543 , 2023

2023

Closest in time.

R. Liu, R. Wu, B. V. Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick, “Zero-1-to-3: Zero-shot one image to 3d object,” 2023

2023

Closest in time.

E. Richardson, G. Metzer, Y. Alaluf, R. Giryes, and D. Cohen-Or, “Texture: Text-guided texturing of 3d shapes,” 2023

2023

Closest in time.

H. Geng, H. Xu, C. Zhao, C. Xu, L. Yi, S. Huang, and H. Wang, “Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts,” 2023

2023

Closest in time.

S. Christen, W. Yang, C. Pérez-D’Arpino, O. Hilliges, D. Fox, and Y.-W. Chao, “Learning human-to-robot handovers from point clouds,” 2023

2023

Closest in time.