2021

A Joint Network for Grasp Detection Conditioned on Natural Language Commands

Chen, Yiye, Xu, Ruinian, Lin, Yunzhi et al.

Understand

We consider the task of grasping a target object based on a natural language command query.

Previous work primarily focused on localizing the object given the query, which requires a separate grasp detection module to grasp it.
The cascaded application of two pipelines incurs errors in overlapping multi-object cases due to ambiguity in the individual outputs.
This work proposes a model named Command Grasping Network(CGNet) to directly output command satisficing grasps from RGB image and textual command inputs.

Built on

C. Ferrari and J. F. Canny, “Planning optimal grasps.” in ICRA , vol. 3, 1992, pp. 2290–2295
1992
Earlier work this paper cites.
C.-P. Tung and A. C. Kak, “Fast construction of force-closure grasps,” IEEE Transactions on Robotics and Automation , vol. 12, no. 4, pp. 615–626, 1996
1996
Earlier work this paper cites.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. Teller, and N. Roy, “Understanding natural language commands for robotic navigation and mobile manipulation,” in Proceedings of the National Conference on Artificial Intelligence (AAAI) , San Francisco, CA, 8 2011, pp. 1507–1514
2011
Earlier work this paper cites.
H. Dang and P. K. Allen, “Semantic grasping: Planning robotic grasps functionally suitable for an object manipulation task,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE, 2012, pp. 1311–1317
2012
Earlier work this paper cites.
J. Krishnamurthy and T. Kollar, “Jointly learning to parse and perceive: Connecting natural language to the physical world,” Transactions of the Association for Computational Linguistics , vol. 1, pp. 193–206, 2013
2013
Earlier work this paper cites.
A. Karpathy, A. Joulin, and L. F. Fei-Fei, “Deep fragment embeddings for bidirectional image sentence mapping,” in Advances in neural information processing systems , 2014, pp. 1889–1897
2014
Earlier work this paper cites.
R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng, “Grounded compositional semantics for finding and describing images with sentences,” Transactions of the Association for Computational Linguistics , vol. 2, pp. 207–218, 2014
2014
Earlier work this paper cites.
S. Guadarrama, E. Rodner, K. Saenko, N. Zhang, R. Farrell, J. Donahue, and T. Darrell, “Open-vocabulary object retrieval.” in Robotics: science and systems , vol. 2, no. 5, 2014, p. 6
2014
Earlier work this paper cites.
I. Lenz, H. Lee, and A. Saxena, “Deep learning for detecting robotic grasps,” The International Journal of Robotics Research , vol. 34, no. 4-5, pp. 705–724, 2015
2015
Earlier work this paper cites.
J. Redmon and A. Angelova, “Real-time grasp detection using convolutional neural networks,” in 2015 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2015, pp. 1316–1322
2015
Earlier work this paper cites.
R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1440–1448
2015
Earlier work this paper cites.

Similar

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015
2015
Cited alongside, same era.
R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell, “Natural language object retrieval,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016
2016
Cited alongside, same era.
A. Rohrbach, M. Rohrbach, R. Hu, T. Darrell, and B. Schiele, “Grounding of textual phrases in images by reconstruction,” in European Conference on Computer Vision . Springer, 2016, pp. 817–834
2016
Cited alongside, same era.
J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” 2017
2017
Cited alongside, same era.
J. Watson, J. Hughes, and F. Iida, “Real-world, real-time robotic grasping with convolutional neural networks,” in Annual Conference Towards Autonomous Robotic Systems . Springer, 2017, pp. 617–626
2017
Cited alongside, same era.
S. Kumra and C. Kanan, “Robotic grasp detection using deep convolutional neural networks,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2017, pp. 769–776
2017
Cited alongside, same era.
E. Jang, S. Vijayanarasimhan, P. Pastor, J. Ibarz, and S. Levine, “End-to-end learning of semantic grasping,” in Conference on Robot Learning , 2017, pp. 119–132
2017
Cited alongside, same era.
J.-H. Kim, K. W. On, W. Lim, J. Kim, J.-W. Ha, and B.-T. Zhang, “Hadamard Product for Low-rank Bilinear Pooling,” in The 5th International Conference on Learning Representations , 2017
2017
Cited alongside, same era.
J. Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, Y. Unno, W. Ko, and J. Tan, “Interactively picking real-world objects with unconstrained spoken language instructions,” in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 3774–3781
2018
Cited alongside, same era.
M. Shridhar and D. Hsu, “Interactive visual grounding of referring expressions for human-robot interaction,” in Proceedings of Robotics: Science and Systems , 2018
2018
Cited alongside, same era.
F.-J. Chu, R. Xu, and P. A. Vela, “Real-world multiobject, multigrasp detection,” IEEE Robotics and Automation Letters , vol. 3, no. 4, pp. 3355–3362, 2018
2018
Cited alongside, same era.

Then

A. B. Rao, K. Krishnan, and H. He, “Learning robotic grasping strategy based on natural-language object descriptions,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2018, pp. 882–887
2018
Later among the works it cites.
D. Huk Park, L. Anne Hendricks, Z. Akata, A. Rohrbach, B. Schiele, T. Darrell, and M. Rohrbach, “Multimodal explanations: Justifying decisions and pointing to the evidence,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 8779–8788
2018
Later among the works it cites.
A. Nguyen, Q. D. Tran, T.-T. Do, I. Reid, D. G. Caldwell, and N. G. Tsagarakis, “Object captioning and retrieval with natural language,” in Proceedings of the IEEE International Conference on Computer Vision Workshops , 2019, pp. 0–0
2019
Later among the works it cites.
H. Zhang, X. Lan, S. Bai, X. Zhou, Z. Tian, and N. Zheng, “Roi-based robotic grasp detection for object overlapping scenes,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2019, pp. 4768–4775
2019
Later among the works it cites.
V. Cohen, B. Burchfiel, T. Nguyen, N. Gopalan, S. Tellex, and G. Konidaris, “Grounding language attributes to objects using bayesian eigenobjects,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2019, pp. 1187–1194
2019
Later among the works it cites.
M. Danielczuk, A. Kurenkov, A. Balakrishna, M. Matl, D. Wang, R. Martín-Martín, A. Garg, S. Savarese, and K. Goldberg, “Mechanical search: Multi-step retrieval of a target object occluded by clutter,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 1614–1621
2019
Later among the works it cites.
T. Nguyen, N. Gopalan, R. Patel, M. Corsaro, E. Pavlick, and S. Tellex, “Robot Object Retrieval with Contextual Natural Language Queries,” in Proceedings of Robotics: Science and Systems , Corvalis, Oregon, USA, July 2020
2020
Later among the works it cites.
A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox, “6-dof grasping for target-driven object manipulation in clutter,” in 2020 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2020, pp. 6232–6238
2020
Later among the works it cites.
W. Liu, A. Daruna, and S. Chernova, “Cage: Context-aware grasping engine,” in 2020 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2020, pp. 2550–2556
2020
Later among the works it cites.
K. Fang, Y. Zhu, A. Garg, A. Kurenkov, V. Mehta, L. Fei-Fei, and S. Savarese, “Learning task-oriented grasping for tool manipulation from simulated self-supervision,” The International Journal of Robotics Research , vol. 39, no. 2-3, pp. 202–216, 2020
2020
Later among the works it cites.
D. Guo, T. Kong, F. Sun, and H. Liu, “Object discovery and grasp detection with a shared convolutional neural network,” in 2016 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2016, pp. 2038–2043
2043
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…