Fetching the paper…

Reading the bibliography…

2024

ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models

Dey, Sombit, Zaech, Jan-Nico, Nikolov, Nikolay et al.

Original
PDF
Post on X

Understand

Recent progress in large language models and access to large-scale robotic datasets has sparked a paradigm shift in robotics models transforming them into generalists able to adapt to various tasks, scenes, and robot modalities.

A large step for the community are open Vision Language Action models which showcase strong performance in a wide variety of tasks.
In this work, we study the visual generalization capabilities of three existing robotic foundation models, and propose a corresponding evaluation framework.
Our study shows that the existing models do not exhibit robustness to visual out-of-domain scenarios.

ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models · Around

Built on

B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Benchmarking in manipulation research: Using the yale-cmu-berkeley object and model set,” IEEE Robotics & Automation Magazine , vol. 22, no. 3, pp. 36–52, 2015
2015
Earlier work this paper cites.
Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox, “Closing the sim-to-real loop: Adapting simulation randomization with real world experience,” in 2019 International Conference on Robotics and Automation (ICRA) , 2019, pp. 8973–8979
2019
Earlier work this paper cites.
F. Xiang, et al. , “SAPIEN: A simulated part-based interactive environment,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 6 2020
2020
Earlier work this paper cites.
W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: a survey,” in 2020 IEEE symposium series on computational intelligence (SSCI) . IEEE, 2020, pp. 737–744
2020
Earlier work this paper cites.
M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox, “ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020
2020
Earlier work this paper cites.
H.-S. Fang, C. Wang, M. Gou, and C. Lu, “Graspnet-1billion: A large-scale benchmark for general object grasping,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020, pp. 11 444–11 453
2020
Earlier work this paper cites.
W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: a survey,” in 2020 IEEE Symposium Series on Computational Intelligence (SSCI) , 2020, pp. 737–744
2020
Earlier work this paper cites.
R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2020
Original
2020
Earlier work this paper cites.
A. Radford, et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763
2021
Earlier work this paper cites.
A. Szot, et al. , “Habitat 2.0: Training home assistants to rearrange their habitat,” in Advances in Neural Information Processing Systems (NeurIPS) , 2021
2021
Earlier work this paper cites.
V. Makoviychuk, et al. , “Isaac gym: High performance gpu-based physics simulation for robot learning,” arXiv preprint arXiv:2108.10470 , 2021
Original
2021
Earlier work this paper cites.
C. Chamzas, C. Quintero-Pena, Z. Kingston, A. Orthey, D. Rakita, M. Gleicher, M. Toussaint, and L. E. Kavraki, “Motionbenchmaker: A tool to generate and benchmark motion planning datasets,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 882–889, 2021
2021
Earlier work this paper cites.
T. Mu, Z. Ling, F. Xiang, D. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su, “Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations,” arXiv preprint arXiv:2107.14483 , 2021
Original
2021
Earlier work this paper cites.

Similar

A. Brohan, et al. , “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817 , 2022
Original
2022
Cited alongside, same era.
J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in ICML , 2022
2022
Cited alongside, same era.
J.-B. Alayrac, et al. , “Flamingo: a visual language model for few-shot learning,” arXiv preprint arXiv:2204.14198 , 2022
Original
2022
Cited alongside, same era.
F. Muratore, F. Ramos, G. Turk, W. Yu, M. Gienger, and J. Peters, “Robot learning from randomized simulations: A review,” Frontiers in Robotics and AI , vol. 9, 2022
2022
Cited alongside, same era.

Then

S. K. Ainsworth, J. Hayase, and S. Srinivasa, “Git re-basin: Merging models modulo permutation symmetries,” arXiv preprint arXiv:2209.04836 , 2023
Original
2023
Later among the works it cites.
Y.-L. Sung, L. Li, K. Lin, Z. Gan, M. Bansal, and L. Wang, “An empirical study of multimodal model merging,” arXiv preprint arXiv:2304.14933 , 2023
Original
2023
Later among the works it cites.
M. Oquab, et al. , “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193 , 2024
Original
2024
Closest in time.
M. J. Kim, et al. , “Openvla: An open-source vision-language-action model,” arXiv preprint arXiv:2406.09246 , 2024
Original
2024
Closest in time.
X. Li, et al. , “Evaluating real-world robot manipulation policies in simulation,” arXiv preprint arXiv:2405.05941 , 2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

D. Arpit, H. Wang, Y. Zhou, and C. Xiong, “Ensemble of averages: Improving model selection and boosting performance in domain generalization,” Advances in Neural Information Processing Systems , vol. 35, pp. 8265–8277, 2022

2022

Cited alongside, same era.

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023, pp. 11 975–11 986

2023

Cited alongside, same era.

Q. Zhang, J. Zhang, Y. Xu, and D. Tao, “Vision transformer with quadrangle attention,” arXiv preprint arXiv:2303.15105 , 2023

2023

Cited alongside, same era.

J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models,” in Proceedings of the 40th International Conference on Machine Learning , ser. ICML’23, 2023

2023

Cited alongside, same era.

A. Padalkar, et al. , “Open x-embodiment: Robotic learning datasets and rt-x models,” arXiv preprint arXiv:2310.08864 , 2023

2023

Cited alongside, same era.

B. Zitkovich, et al. , “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” in Proceedings of The 7th Conference on Robot Learning , ser. Proceedings of Machine Learning Research, J. Tan, M. Toussaint, and K. Darvish, Eds., vol. 229. PMLR, 11 2023, pp. 2165–2183

2023

Cited alongside, same era.

X. Chen, et al. , “Pali-x: On scaling up a multilingual vision and language model,” arXiv preprint arXiv:2305.18565 , 2023

2023

Cited alongside, same era.

D. Driess, et al. , “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378 , 2023

2023

Cited alongside, same era.

2024

Closest in time.

Octo Model Team, et al. , “Octo: An open-source generalist robot policy,” in Proceedings of Robotics: Science and Systems , Delft, Netherlands, 2024

2024

Closest in time.

S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh, “Prismatic vlms: Investigating the design space of visually-conditioned language models,” arXiv preprint arXiv:2402.07865 , 2024

2024

Closest in time.

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, 2024

2024

Closest in time.

L. Beyer, et al. , “PaliGemma: A versatile 3B VLM for transfer,” arXiv preprint arXiv:2407.07726 , 2024

2024

Closest in time.

Z. Ren, Z. Huang, Y. Wei, Y. Zhao, D. Fu, J. Feng, and X. Jin, “Pixellm: Pixel reasoning with large multimodal model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2024, pp. 26 374–26 383

2024

Closest in time.

E. Yang, L. Shen, G. Guo, X. Wang, X. Cao, J. Zhang, and D. Tao, “Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities,” arXiv preprint arXiv:2408.07666 , 2024

2024

Closest in time.

A. Alexandrov, V. Raychev, M. N. Müller, C. Zhang, M. Vechev, and K. Toutanova, “Mitigating catastrophic forgetting in language transfer via model merging,” arXiv preprint arXiv:2407.08699 , 2024

2024

Closest in time.