Fetching the paper…

Reading the bibliography…

VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation · Around

2022

VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation

Zheng, Kaizhi, Chen, Xiaotong, Jenkins, Odest Chadwicke et al.

Original
PDF
Post on X

Understand

Benefiting from language flexibility and compositionality, humans naturally intend to use language to command an embodied agent for complex tasks such as navigation and object manipulation.

In this work, we aim to fill the blank of the last mile of embodied agents -- object manipulation by following human guidance, e.g., "move the red mug next to the box while keeping it upright." To this end, we introduce an Automatic Manipulation Solver (AMSolver) system and build a Vision-and-Language Manipulation benchmark (VLMbench) based on it, containing various language instructions on categorized robotic manipulation tasks.
Specifically, modular rule-based task templates are created to automatically generate robot demonstrations with language instructions, consisting of diverse object shapes and appearances, action types, and motion constraints.
We also develop a keypoint-based model 6D-CLIPort to deal with multi-view observations and language input and output a sequence of 6 degrees of freedom (DoF) actions.

Built on

Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., Fox, D.: ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2020), https://arxiv.org/abs/1912.01734
Original
1912
Earlier work this paper cites.
Şucan, I.A., Moll, M., Kavraki, L.E.: The Open Motion Planning Library. IEEE Robotics & Automation Magazine 19
2012
Earlier work this paper cites.
Rohmer, E., Singh, S.P.N., Freese, M.: V-rep: A versatile and scalable robot simulation framework pp. 1321–1326 (2013). https://doi.org/10.1109/IROS.2013.6696520
2013
Earlier work this paper cites.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: Visual question answering pp. 2425–2433 (2015)
2015
Earlier work this paper cites.
Gao, L., Guo, Z., Zhang, H., Xu, X., Shen, H.T.: Video captioning with attention-based lstm and semantic consistency. IEEE Transactions on Multimedia 19
2017
Earlier work this paper cites.
ten Pas, A., Gualtieri, M., Saenko, K., Platt, R.: Grasp pose detection in point clouds. The International Journal of Robotics Research 36
2017
Earlier work this paper cites.
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., Van Den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments pp. 3674–3683 (2018)
2018
Earlier work this paper cites.
Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., Batra, D.: Embodied question answering pp. 1–10 (2018)
2018
Earlier work this paper cites.
Wang, B., Ma, L., Zhang, W., Liu, W.: Reconstruction network for video captioning pp. 7622–7631 (2018)
2018
Earlier work this paper cites.
James, S., Freese, M., Davison, A.J.: Pyrep: Bringing v-rep to deep robot learning. arXiv preprint arXiv:1906.11176 (2019)
Original
2019
Earlier work this paper cites.
Liu, R., Liu, C., Bai, Y., Yuille, A.L.: Clevr-ref+: Diagnosing visual reasoning with referring expressions pp. 4185–4194 (2019)
2019
Earlier work this paper cites.
Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems 32
2019
Earlier work this paper cites.
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., Tenenbaum, J.B.: Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442 (2019)
Original
2019
Earlier work this paper cites.

Similar

Ahmed, O., Träuble, F., Goyal, A., Neitz, A., Bengio, Y., Schölkopf, B., Wüthrich, M., Bauer, S.: Causalworld: A robotic manipulation benchmark for causal structure and transfer learning. arXiv preprint arXiv:2010.04296 (2020)
Original
2020
Cited alongside, same era.
Chen, H., Ding, G., Liu, X., Lin, Z., Liu, J., Han, J.: Imram: Iterative matching with recurrent attention memory for cross-modal image-text retrieval pp. 12655–12663 (2020)
2020
Cited alongside, same era.
James, S., Ma, Z., Arrojo, D.R., Davison, A.J.: Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters 5
2020
Cited alongside, same era.
Kurita, S., Cho, K.: Generative language-grounded policy in vision-and-language navigation with bayes’ rule. arXiv preprint arXiv:2009.07783 (2020)
Original
2020
Cited alongside, same era.
Lynch, C., Sermanet, P.: Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648 (2020)

Then

Liu, W., Paxton, C., Hermans, T., Fox, D.: Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects. arXiv preprint arXiv:2110.10189 (2021)
Original
2021
Later among the works it cites.
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., Martín-Martín, R.: What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298 (2021)
Original
2021
Later among the works it cites.
Pashevich, A., Schmid, C., Sun, C.: Episodic transformer for vision-and-language navigation pp. 15942–15952 (2021)
2021
Later among the works it cites.
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision pp. 8748–8763 (2021)
2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2020

Cited alongside, same era.

Stepputtis, S., Campbell, J., Phielipp, M., Lee, S., Baral, C., Amor, H.B.: Language-conditioned imitation learning for robot manipulation tasks (2020)

2020

Cited alongside, same era.

Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., Levine, S.: Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning pp. 1094–1100 (2020)

2020

Cited alongside, same era.

Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J., Attarian, M., Armstrong, T., Krasin, I., Duong, D., Sindhwani, V., Lee, J.: Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL) (2020)

2020

Cited alongside, same era.

Zhang, Q., Lei, Z., Zhang, Z., Li, S.Z.: Context-aware attention network for image-text retrieval pp. 3536–3545 (2020)

2020

Cited alongside, same era.

Zhu, Y., Wong, J., Mandlekar, A., Martín-Martín, R.: robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293 (2020)

2020

Cited alongside, same era.

Ehsani, K., Han, W., Herrasti, A., VanderBilt, E., Weihs, L., Kolve, E., Kembhavi, A., Mottaghi, R.: Manipulathor: A framework for visual object manipulation pp. 4497–4506 (2021)

2021

Cited alongside, same era.

Goyal, P., Mooney, R.J., Niekum, S.: Zero-shot task adaptation using natural language. arXiv preprint arXiv:2106.02972 (2021)

2021

Cited alongside, same era.

Shao, L., Migimatsu, T., Zhang, Q., Yang, K., Bohg, J.: Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research 40

2021

Later among the works it cites.

Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., Maestre, N., Mukadam, M., Chaplot, D.S., Maksymets, O., et al.: Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems 34

2021

Later among the works it cites.

Zhang, H., Lu, Y., Yu, C., Hsu, D., La, X., Zheng, N.: Invigorate: Interactive visual grounding and grasping in clutter. arXiv preprint arXiv:2108.11092 (2021)

2021

Later among the works it cites.

Mees, O., Hermann, L., Burgard, W.: What matters in language conditioned robotic imitation learning. arXiv preprint arXiv:2204.06252 (2022)

2022

Closest in time.

Mees, O., Hermann, L., Rosete-Beas, E., Burgard, W.: Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L) 7

2022

Closest in time.

Shridhar, M., Manuelli, L., Fox, D.: Cliport: What and where pathways for robotic manipulation pp. 894–906 (2022)

2022

Closest in time.

Srivastava, S., Li, C., Lingelbach, M., Martín-Martín, R., Xia, F., Vainio, K.E., Lian, Z., Gokmen, C., Buch, S., Liu, K., et al.: Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments pp. 477–490 (2022)

2022

Closest in time.

Yuan, W., Paxton, C., Desingh, K., Fox, D.: Sornet: Spatial object-centric representations for sequential manipulation pp. 148–157 (2022)

2022

Closest in time.