Fetching the paper…
Reading the bibliography…
Comprehensive, unbiased, and comparable evaluation of modern generalist policies is uniquely challenging: existing approaches for robot benchmarking typically rely on heavy standardization, either by specifying fixed evaluation tasks and environments, or by hosting centralized ''robot challenges'', and do not readily scale to evaluating generalist policies across a broad range of tasks and environments.
Die berechnung der turnier-ergebnisse als ein maximumproblem der wahrscheinlichkeitsrechnung
E. Zermelo · 1929
Earlier work this paper cites.
Rank analysis of incomplete block designs: I. the method of paired comparisons
R. A. Bradley and M. E. Terry · 1952
Earlier work this paper cites.
The proposed uscf rating system, its development, theory, and applications
A. E. Elo · 1967
Earlier work this paper cites.
On extending the bradley-terry model to accommodate ties in paired comparison experiments
R. R. Davidson · 1970
Earlier work this paper cites.
Robocup: A challenge problem for ai
H. Kitano, M. Asada, Y. Kuniyoshi, I. Noda, E. Osawa, and H. Matsubara · 1997
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
E. Todorov, T. Erez, and Y. Tassa · 2012
Earlier work this paper cites.
The ycb object and model set: Towards common benchmarks for manipulation research
B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar · 2015
Earlier work this paper cites.
Analysis and observations from the first amazon picking challenge
N. Correll, K. E. Bekris, D. Berenson, O. Brock, A. Causo, K. Hauser, K. Okada, A. Rodriguez, J. M. Romano, and P. R. Wurman · 2016
Earlier work this paper cites.
Ai2-thor: An interactive 3d environment for visual ai
E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, et al · 2017
Earlier work this paper cites.
The robotarium: A remotely accessible swarm robotics research testbed
D. Pickem, P. Glotfelter, L. Wang, M. Mote, A. Ames, E. Feron, and M. Egerstedt · 2017
Earlier work this paper cites.
The darpa robotics challenge finals: Results and perspectives
E. Krotkov, D. Hackett, L. Jackel, M. Perschbacher, J. Pippine, J. Strauss, G. Pratt, and C. Orlowski · 2018
Earlier work this paper cites.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al · 2018
Earlier work this paper cites.
Robot learning in homes: Improving generalization and reducing dataset bias
A. Gupta, A. Murali, D. P. Gandhi, and L. Pinto · 2018
Earlier work this paper cites.
Replab: A reproducible low-cost arm benchmark for robotic learning
B. Yang, D. Jayaraman, J. Zhang, and S. Levine · 2019
Earlier work this paper cites.
Robonet: Large-scale multi-robot learning
S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn · 2019
Earlier work this paper cites.
Rlbench: The robot learning benchmark & learning environment
S. James, Z. Ma, D. R. Arrojo, and A. J. Davison · 2020
Earlier work this paper cites.
IKEA furniture assembly environment for long-horizon complex manipulation tasks
Y. Lee, E. S. Hu, Z. Yang, A. Yin, and J. J. Lim · 2021
Earlier work this paper cites.
Bridge data: Boosting generalization of robotic skills with cross-domain datasets
F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine · 2021
Earlier work this paper cites.
Real robot challenge: A robotics competition in the cloud
S. Bauer, M. Wüthrich, F. Widmaier, A. Buchholz, S. Stark, A. Goyal, T. Steinbrenner, J. Akpo, S. Joshi, V. Berenz, et al · 2022
Earlier work this paper cites.
Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks
O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard · 2022
Earlier work this paper cites.
Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments
S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K. E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu, et al · 2022
Earlier work this paper cites.
Rb2: Robotic manipulation benchmarking with a twist
S. Dasari, J. Wang, J. Hong, S. Bahl, Y. Lin, A. Wang, A. Thankaraj, K. Chahal, B. Calli, S. Gupta, et al · 2022
Earlier work this paper cites.
Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation
M. Heo, Y. Lee, D. Lee, and J. J. Lim · 2023
Earlier work this paper cites.
FMB: A functional manipulation benchmark for generalizable robotic learning
J. Luo, C. Xu, F. Liu, L. Tan, Z. Lin, J. Wu, P. Abbeel, and S. Levine · 2023
Earlier work this paper cites.
Train offline, test online: A real robot learning benchmark, 2023
G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, J. Pari, K. Hatch, A. Jain, T. Yu, P. Abbeel, L. Pinto, C. Finn, and A. Gupta · 2023
Cited alongside, same era.
Libero: Benchmarking knowledge transfer for lifelong robot learning
B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone · 2023
Cited alongside, same era.
Dexart: Benchmarking generalizable dexterous manipulation with articulated objects
C. Bao, H. Xu, Y. Qin, and X. Wang · 2023
Cited alongside, same era.
Rmbench: Benchmarking deep reinforcement learning for robotic manipulator control
Y. Xiang, X. Wang, S. Hu, B. Zhu, X. Huang, X. Wu, and S. Lyu · 2023
Cited alongside, same era.
Ramp: A benchmark for evaluating robotic assembly manipulation and planning
J. Collins, M. Robson, J. Yamada, M. Sridharan, K. Janik, and I. Posner · 2023
Cited alongside, same era.
p i _ 0 pi\_0 : A vision-language-action flow model for general robot control
K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al · 2024
Later among the works it cites.
Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation
J. Wen, Y. Zhu, J. Li, M. Zhu, K. Wu, Z. Xu, N. Liu, R. Cheng, C. Shen, Y. Peng, F. Feng, and J. Tang · 2024
Later among the works it cites.
3d-vla: 3d vision-language-action generative world model
H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y. Du, Y. Hong, and C. Gan · 2024
Later among the works it cites.
Minivla: A better vla with a smaller footprint, 2024
S. Belkhale and D. Sadigh · 2024
Later among the works it cites.
From multimodal llms to generalist embodied agents: Methods and lessons
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al · 2023
Cited alongside, same era.
Rt-2: Vision-language-action models transfer web knowledge to robotic control
A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich · 2023
Cited alongside, same era.
BridgeData v2: A dataset for robot learning at scale
H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, et al · 2023
Cited alongside, same era.
N. M. M. Shafiullah, A. Rai, H. Etukuru, Y. Liu, I. Misra, S. Chintala, and L. Pinto · 2023
Cited alongside, same era.
Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models
N. Guha, J. Nyarko, D. Ho, C. Ré, A. Chilton, A. Chohlas-Wood, A. Peters, B. Waldon, D. Rockmore, D. Zambrano, et al · 2023
Cited alongside, same era.
Finite scalar quantization: Vq-vae made simple, 2023
F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen · 2023
Cited alongside, same era.
Diffusion policy: Visuomotor policy learning via action diffusion
C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song · 2023
Cited alongside, same era.
A. Szot, B. Mazoure, O. Attia, A. Timofeev, H. Agrawal, D. Hjelm, Z. Gan, Z. Kira, and A. Toshev · 2024
Later among the works it cites.
Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking
H. Bharadhwaj, J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar · 2024
Later among the works it cites.
Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot
H.-S. Fang, H. Fang, Z. Tang, J. Liu, C. Wang, J. Wang, H. Zhu, and C. Lu · 2024
Later among the works it cites.
Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots
C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song · 2024
Later among the works it cites.
Robot learning as an empirical science: Best practices for policy evaluation
H. Kress-Gazit, K. Hashimoto, N. Kuppuswamy, P. Shah, P. Horgan, G. Richardson, S. Feng, and B. Burchfiel · 2024
Later among the works it cites.
Paligemma: A versatile 3b vlm for transfer
L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, et al · 2024
Later among the works it cites.
Robotic control via embodied chain-of-thought reasoning
M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine · 2024
Later among the works it cites.
Behavior generation with latent actions
S. Lee, Y. Wang, H. Etukuru, H. J. Kim, N. M. M. Shafiullah, and L. Pinto · 2024
Later among the works it cites.
Fast: Efficient action tokenization for vision-language-action models
K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine · 2025
Closest in time.
Gemini robotics: Bringing ai into the physical world
G. R. Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al · 2025
Closest in time.
Autoeval: Autonomous evaluation of generalist robot manipulation policies in the real world
Z. Zhou, P. Atreya, Y. L. Tan, K. Pertsch, and S. Levine · 2025
Closest in time.
The Earth Rover Challenge
The Earth Rover Challenge Organizers · 2025
Closest in time.
Dexvla: Vision-language model with plug-in diffusion expert for general robot control
J. Wen, Y. Zhu, J. Li, Z. Tang, C. Shen, and F. Feng · 2025
Closest in time.
Gr00t n1: An open foundation model for generalist humanoid robots
J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al · 2025
Closest in time.
AgiBot-World-Contributors, Q. Bu, J. Cai, L. Chen, X. Cui, Y. Ding, S. Feng, S. Gao, X. He, X. Hu, X. Huang, S. Jiang, Y. Jiang, C. Jing, H. Li, J. Li, C. Liu, Y. Liu, Y. Lu, J. Luo, P. Luo, Y. Mu, Y. Niu, Y. Pan, J. Pang, Y. Qiao, G. Ren, C. Ruan, J. Shan, Y. Shen, C. Shi, M. Shi, M. Shi, C. Sima, J. Song, H. Wang, W. Wang, D. Wei, C. Xie, G. Xu, J. Yan, C. Yang, L. Yang, S. Yang, M. Yao, J. Zeng, C. Zhang, Q. Zhang, B. Zhao, C. Zhao, J. Zhao, and J. Zhu · 2025
Closest in time.
Open pi zero: An open source raspberry pi zero
A. Z. Ren · 2025
Closest in time.
pi0.5: a vision-language-action model with open-world generalization
P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al · 2025
Closest in time.
A taxonomy for evaluating generalist robot policies
J. Gao, S. Belkhale, S. Dasari, A. Balakrishna, D. Shah, and D. Sadigh · 2025
Closest in time.
Regarding Meta, Llama 4, and Cheating, 2025
The Register · 2025
Closest in time.