2022

OpenD: A Benchmark for Language-Driven Door and Drawer Opening

Zhao, Yizhou, Gao, Qiaozi, Qiu, Liang et al.

Understand

We introduce OPEND, a benchmark for learning how to use a hand to open cabinet doors or drawers in a photo-realistic and physics-reliable simulation environment driven by language instruction.

To solve the task, we propose a multi-step planner composed of a deep neural network and rule-base controllers.
The network is utilized to capture spatial relationships from images and understand semantic meaning from language instructions.
Controllers efficiently execute the plan based on the spatial and semantic understanding.

Built on

M. Stilman, “Global manipulation planning in robot joint space with task constraints,” IEEE Transactions on Robotics
2010
Earlier work this paper cites.
J.-P. Saut and D. Sidobre, “Efficient models for grasp planning with a multi-fingered hand,” Robotics and Autonomous Systems
2012
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision
2014
Earlier work this paper cites.
V. Kumar and E. Todorov, “Mujoco haptix: A virtual reality system for hand manipulation,” in 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids)
2015
Earlier work this paper cites.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in neural information processing systems
2015
Earlier work this paper cites.
E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, “Ai2-thor: An interactive 3d environment for visual ai,” arXiv preprint arXiv:1712.05474
Original
2017
Earlier work this paper cites.
R. T. Arrazate, “Development of a urdf file for simulation and programming of a delta robot using ros,” Santiago de Querétaro
2017
Earlier work this paper cites.
X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba, “Virtualhome: Simulating household activities via programs,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
2018
Earlier work this paper cites.
M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox, “Alfred: A benchmark for interpreting grounded instructions for everyday tasks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
2020
Earlier work this paper cites.
X. Lin, Y. Wang, J. Olkin, and D. Held, “Softgym: Benchmarking deep reinforcement learning for deformable object manipulation,” arXiv preprint arXiv:2011.07215
Original
2020
Earlier work this paper cites.

Similar

C. Lynch and P. Sermanet, “Language conditioned imitation learning over unstructured data,” arXiv preprint arXiv:2005.07648
Original
2020
Cited alongside, same era.
F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al
2020
Cited alongside, same era.
S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643
Original
2020
Cited alongside, same era.
T. Mu, Z. Ling, F. Xiang, D. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su, “Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations,” arXiv preprint arXiv:2107.14483
Original
2021
Cited alongside, same era.
K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mottaghi, “Manipulathor: A framework for visual object manipulation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
2021
Cited alongside, same era.
A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín, “What matters in learning from offline human demonstrations for robot manipulation,” in arXiv preprint arXiv:2108.03298
Original
2021
Cited alongside, same era.
A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al
2021
Cited alongside, same era.
C. Li, F. Xia, R. Martín-Martín, M. Lingelbach, S. Srivastava, B. Shen, K. Vainio, C. Gokmen, G. Dharan, T. Jain, et al
Original
2021
Cited alongside, same era.
A. Suglia, Q. Gao, J. Thomason, G. Thattai, and G. Sukhatme, “Embodied bert: A transformer model for embodied, language-guided visual task completion,” arXiv preprint arXiv:2108.04927
Original
2021
Cited alongside, same era.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al
2021
Cited alongside, same era.

Then

T. Tran and C. Ekenna, “Identifying valid robot configurations via a deep learning approach,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
2021
Later among the works it cites.
J. Duan, S. Yu, H. L. Tan, H. Zhu, and C. Tan, “A survey of embodied ai: From simulators to research tasks,” IEEE Transactions on Emerging Topics in Computational Intelligence
2022
Closest in time.
K. Zheng, X. Chen, O. C. Jenkins, and X. E. Wang, “Vlmbench: A compositional benchmark for vision-and-language manipulation,” arXiv preprint arXiv:2206.08522
Original
2022
Closest in time.
S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K. E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu, et al
2022
Closest in time.
M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in Conference on Robot Learning
2022
Closest in time.
O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, “Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,” IEEE Robotics and Automation Letters
2022
Closest in time.
O. Mees, L. Hermann, and W. Burgard, “What matters in language conditioned robotic imitation learning,” arXiv preprint arXiv:2204.06252
Original
2022
Closest in time.
X. Gao, Q. Gao, R. Gong, K. Lin, G. Thattai, and G. S. Sukhatme, “Dialfred: Dialogue-enabled agents for embodied instruction following,” arXiv preprint arXiv:2202.13330
Original
2022
Closest in time.
Y. Narang, K. Storey, I. Akinola, M. Macklin, P. Reist, L. Wawrzyniak, Y. Guo, A. Moravanszky, G. State, M. Lu, et al
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…