MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulation

I. A. Şucan, M. Moll, and L. E. Kavraki, “The Open Motion Planning Library,” IEEE Robotics & Automation Magazine , vol. 19, no. 4, pp. 72–82, December 2012, https://ompl.kavrakilab.org

2012

Earlier work this paper cites.

C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 652–660

2017

Earlier work this paper cites.

D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke et al. , “QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation,” arXiv preprint arXiv:1806.10293 , 2018

Original

2018

Earlier work this paper cites.

A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei, “RoboTurk: A crowdsourcing platform for robotic skill learning through imitation,” CoRR , vol. abs/1811.02790, 2018

Original

2018

Earlier work this paper cites.

S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3019–3026, 2020

2020

Earlier work this paper cites.

J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502 , 2020

Original

2020

Earlier work this paper cites.

S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn, “Robonet: Large-scale multi-robot learning,” in Conference on Robot Learning . PMLR, 2020, pp. 885–897

2020

Earlier work this paper cites.

Y. Pang, W. Wang, F. E. H. Tay, W. Liu, Y. Tian, and L. Yuan, “Masked autoencoders for point cloud self-supervised learning,” 2022. [Online]. Available: https://arxiv.org/abs/2203.06604

Original

2022

Earlier work this paper cites.

Y. Zhu, P. Stone, and Y. Zhu, “Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 4126–4133, 2022

2022

Earlier work this paper cites.

S. Nasiriany, T. Gao, A. Mandlekar, and Y. Zhu, “Learning and retrieval from prior data for skill-based imitation learning,” in Conference on Robot Learning (CoRL) , 2022

2022

Earlier work this paper cites.

E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” in Conference on Robot Learning . PMLR, 2022, pp. 991–1002

2022

Earlier work this paper cites.

F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, “Bridge data: Boosting generalization of robotic skills with cross-domain datasets,” in RSS , 2022

2022

Earlier work this paper cites.

A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz et al. , “Rt-1: Robotics transformer for real-world control at scale,” in arXiv preprint arXiv:2212.06817 , 2022

Original

2022

Earlier work this paper cites.

Z. J. Cui, Y. Wang, N. M. M. Shafiullah, and L. Pinto, “From play to policy: Conditional behavior generation from uncurated robot data,” arXiv preprint arXiv:2210.10047 , 2022

Original

2022

Earlier work this paper cites.

E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard, “Latent plans for task agnostic offline reinforcement learning,” in Proceedings of the 6th Conference on Robot Learning (CoRL) , 2022

2022

Earlier work this paper cites.

Open X-Embodiment Collaboration, A. Padalkar, A. Pooley et al. , “Open X-Embodiment: Robotic learning datasets and RT-X models,” https://arxiv.org/abs/2310.08864 , 2023

Original

2023

Earlier work this paper cites.

D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al. , “Palm-e: an embodied multimodal language model,” in Proceedings of the 40th International Conference on Machine Learning , 2023, pp. 8469–8488

2023

Earlier work this paper cites.

C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proceedings of Robotics: Science and Systems (RSS) , 2023

2023

Earlier work this paper cites.

H. Wu, Y. Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong, “Unleashing large-scale video generative pre-training for visual robot manipulation,” arXiv preprint arXiv:2312.13139 , 2023

Original

2023

Earlier work this paper cites.

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in International Conference on Computer Vision (ICCV) , 2023

2023

Earlier work this paper cites.

H. Liu, S. Nasiriany, L. Zhang, Z. Bao, and Y. Zhu, “Robot learning on the job: Human-in-the-loop autonomy and learning during deployment,” in Robotics: Science and Systems (RSS) , 2023

2023

Earlier work this paper cites.

H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruch, Q. Vuong, A. He, V. Myers, K. Fang, C. Finn, and S. Levine, “Bridgedata v2: A dataset for robot learning at scale,” 2023

2023

Earlier work this paper cites.

R. Mendonca, S. Bahl, and D. Pathak, “Structured world models from human videos,” CoRL , 2023

2023

Cited alongside, same era.

A. Padalkar, G. Quere, A. Raffin, J. Silvério, and F. Stulp, “A guided reinforcement learning approach using shared control templates for learning manipulation skills in the real world,” 2023

2023

Cited alongside, same era.

N. M. M. Shafiullah, A. Rai, H. Etukuru, Y. Liu, I. Misra, S. Chintala, and L. Pinto, “On bringing robots home,” 2023

2023

Cited alongside, same era.

J. Luo, C. Xu, F. Liu, L. Tan, Z. Lin, J. Wu, P. Abbeel, and S. Levine, “FMB: A functional manipulation benchmark for generalizable robotic learning,” https://functional-manipulation-benchmark.github.io , 2023

2023

Cited alongside, same era.

M. Heo, Y. Lee, D. Lee, and J. J. Lim, “Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation,” in Robotics: Science and Systems , 2023

Y. Hu, Y. Guo, P. Wang, X. Chen, Y.-J. Wang, J. Zhang, K. Sreenath, C. Lu, and J. Chen, “Video prediction policy: A generalist robot policy with predictive visual representations,” arXiv preprint arXiv:2412.14803 , 2024

Original

2024

Later among the works it cites.

S. Zhou, Y. Du, J. Chen, Y. Li, D.-Y. Yeung, and C. Gan, “Robodreamer: Learning compositional world models for robot imagination,” arXiv preprint arXiv:2404.12377 , 2024

Original

2024

Later among the works it cites.

X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu et al. , “Emu3: Next-token prediction is all you need,” arXiv preprint arXiv:2409.18869 , 2024

Original

2024

Later among the works it cites.

P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel, “Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators,” 2024. [Online]. Available: https://arxiv.org/abs/2309.13037

Original

2024

Later among the works it cites.