2021

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

Liu, Weiyu, Paxton, Chris, Hermans, Tucker et al.

Understand

Geometric organization of objects into semantically meaningful arrangements pervades the built world.

As such, assistive robots operating in warehouses, offices, and homes would greatly benefit from the ability to recognize and rearrange objects into these semantically meaningful structures.
To be useful, these robots must contend with previously unseen objects and receive instructions without significant programming.
While previous works have examined recognizing pairwise semantic relations and sequential manipulation to change these simple relations none have shown the ability to arrange objects into complex structures such as circles or table settings.

Built on

M. Wilson and T. Hermans, “Learning to Manipulate Object Collections Using Grounded State Representations,” in Conference on Robot Learning (CoRL) , 2019. [Online]. Available: https://arxiv.org/abs/1909.07876
Original
1909
Earlier work this paper cites.
J. J. Kuffner and S. M. LaValle, “RRT-connect: An efficient approach to single-query path planning,” in Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065) , vol. 2. IEEE, 2000, pp. 995–1001
2000
Earlier work this paper cites.
B. Rosman and S. Ramamoorthy, “Learning spatial relationships between objects,” The International Journal of Robotics Research , vol. 30, no. 11, pp. 1328–1342, 2011
2011
Earlier work this paper cites.
Y. Jiang, M. Lim, and A. Saxena, “Learning object arrangements in 3d scenes using human context,” in Proceedings of the 29th International Coference on International Conference on Machine Learning . Omnipress, 2012, p. 907–914
2012
Earlier work this paper cites.
M. Cakmak and L. Takayama, “Towards a comprehensive chore list for domestic robots,” IEEE International Conference on Human-Robot Interaction (HRI) , pp. 93–94, 2013
2013
Earlier work this paper cites.
L. Kunze, C. Burbridge, M. Alberti, A. Thippur, J. Folkesson, P. Jensfelt, and N. Hawes, “Combining top-down spatial reasoning and bottom-up object class recognition for scene understanding,” in IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE, 2014, pp. 2910–2915
2014
Earlier work this paper cites.
S. Fichtl, A. McManus, W. Mustafa, D. Kraft, N. Krüger, and F. Guerin, “Learning spatial relationships from 3d vision using histograms,” in IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2014, pp. 501–508
2014
Earlier work this paper cites.
R. Paul, J. Arkin, N. Roy, and T. Howard, “Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators,” in Robotics: Science and Systems , 2016
2016
Earlier work this paper cites.
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma et al. , “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” International journal of computer vision , vol. 123, no. 1, pp. 32–73, 2017
2017
Earlier work this paper cites.
O. Mees, N. Abdo, M. Mazuran, and W. Burgard, “Metric learning for generalizing spatial relations to new objects,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2017, pp. 3175–3182
2017
Earlier work this paper cites.
J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick, “Clevr: A diagnostic dataset for compositional language and elementary visual reasoning,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 2901–2910
2017
Earlier work this paper cites.
A. Suhr, M. Lewis, J. Yeh, and Y. Artzi, “A corpus of natural language for visual reasoning,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 2017, pp. 217–223
2017
Earlier work this paper cites.
J. Li, K. Xu, S. Chaudhuri, E. Yumer, H. Zhang, and L. Guibas, “Grass: Generative recursive autoencoders for shape structures,” ACM Transactions on Graphics (TOG) , vol. 36, no. 4, pp. 1–14, 2017
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems , 2017, pp. 5998–6008
2017
Earlier work this paper cites.
E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation in robotics, games and machine learning,” 2017
2017
Earlier work this paper cites.
M. Günther, J. Ruiz-Sarmiento, C. Galindo, J. González-Jiménez, and J. Hertzberg, “Context-aware 3d object anchoring for mobile robots,” Robotics and Autonomous Systems , vol. 110, pp. 12–32, 2018
2018
Earlier work this paper cites.
M. Janner, K. Narasimhan, and R. Barzilay, “Representation learning for grounded spatial reasoning,” Transactions of the Association for Computational Linguistics , vol. 6, pp. 49–61, 2018
2018
Earlier work this paper cites.
Y. Bisk, K. J. Shih, Y. Choi, and D. Marcu, “Learning interpretable spatial operations in a rich 3d blocks world,” in Thirty-Second AAAI Conference on Artificial Intelligence , 2018
2018
Earlier work this paper cites.
Z. Zeng, Z. Zhou, Z. Sui, and O. C. Jenkins, “Semantic robot programming for goal-directed manipulation in cluttered scenes,” in IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 7462–7469
2018
Earlier work this paper cites.

Similar

K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and J. B. Tenenbaum, “Neural-symbolic vqa: Disentangling reasoning from vision and language understanding,” in Advances in Neural Information Processing Systems , 2018, pp. 1039–1050
2018
Cited alongside, same era.
D. Barrett, F. Hill, A. Santoro, A. Morcos, and T. Lillicrap, “Measuring abstract reasoning in neural networks,” in International conference on machine learning . PMLR, 2018, pp. 511–520
2018
Cited alongside, same era.
P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas, “Learning representations and generative models for 3d point clouds,” in International conference on machine learning . PMLR, 2018, pp. 40–49
2018
Cited alongside, same era.
C. Paxton, Y. Bisk, J. Thomason, A. Byravan, and D. Foxl, “Prospection: Interpretable plans from language by predicting the future,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 6942–6948
2019
Cited alongside, same era.
C. Zhang, F. Gao, B. Jia, Y. Zhu, and S.-C. Zhu, “Raven: A dataset for relational and analogical visual reasoning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 5317–5327
2019
Cited alongside, same era.
J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 165–174
2019
Cited alongside, same era.
K. Mo, P. Guerrero, L. Yi, H. Su, P. Wonka, N. Mitra, and L. Guibas, “Structurenet: Hierarchical graph networks for 3d shape generation,” ACM Transactions on Graphics (TOG), Siggraph Asia 2019 , vol. 38, no. 6, p. Article 242, 2019
2019
Cited alongside, same era.
M. Li, A. G. Patil, K. Xu, S. Chaudhuri, O. Khan, A. Shamir, C. Tu, B. Chen, D. Cohen-Or, and H. Zhang, “Grains: Generative recursive autoencoders for indoor scenes,” ACM Transactions on Graphics (TOG) , vol. 38, no. 2, pp. 1–16, 2019
2019
Cited alongside, same era.
Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, “On the continuity of rotation representations in neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 5745–5753
2019
Cited alongside, same era.
A. Mousavian, C. Eppner, and D. Fox, “6-DOF graspnet: Variational grasp generation for object manipulation,” International Conference on Computer Vision , pp. 2901–2910, 2019
2019
Cited alongside, same era.
D. Batra, A. Chang, S. Chernova, A. Davison, J. Deng, V. Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi, M. Savva, and H. Su, “Rearrangement: A challenge for embodied ai,” ArXiv , vol. abs/2011.01975, 2020
Original
2020
Cited alongside, same era.
S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots that use language,” Annual review of control, robotics, and autonomous systems , no. 3, 2020
2020
Cited alongside, same era.

Then

Y. Hristov, D. Angelov, M. Burke, A. Lascarides, and S. Ramamoorthy, “Disentangled relational representations for explaining and learning from demonstration,” in Conference on Robot Learning . PMLR, 2020, pp. 870–884
2020
Later among the works it cites.
D. Ding, F. Hill, A. Santoro, and M. Botvinick, “Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures,” arXiv preprint arXiv:2012.08508 , 2020
Original
2020
Later among the works it cites.
M. Nazarczuk and K. Mikolajczyk, “Shop-vrb: A visual reasoning benchmark for object perception,” in 2020 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2020, pp. 6898–6904
2020
Later among the works it cites.
R. Girdhar and D. Ramanan, “CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning,” in ICLR , 2020
2020
Later among the works it cites.
K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum, “CLEVRER: collision events for video representation and reasoning,” in ICLR , 2020
2020
Later among the works it cites.
R. Wu, Y. Zhuang, K. Xu, H. Zhang, and B. Chen, “Pq-net: A generative part seq2seq network for 3d shapes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 829–838
2020
Later among the works it cites.
C. Eppner, A. Mousavian, and D. Fox, “ACRONYM: A large-scale grasp dataset based on simulation,” in IEEE Int. Conf. on Robotics and Automation, ICRA , 2020
2020
Later among the works it cites.
X. Wang, C. Yeshwanth, and M. Nießner, “SceneFormer: Indoor scene generation with transformers,” arXiv preprint arXiv:2012.09793 , 2020
Original
2020
Later among the works it cites.
G. Radevski, G. Collell, M. F. Moens, and T. Tuytelaars, “Decoding language spatial relations to 2d spatial arrangements,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings , 2020, pp. 4549–4560
2020
Later among the works it cites.
C. Paxton, C. Xie, T. Hermans, and D. Fox, “Predicting stable configurations for semantic placement of novel objects,” in Conference on Robot Learning (CoRL) , 2021
2021
Closest in time.
S. G. Venkatesh, A. Biswas, R. Upadrashta, V. Srinivasan, P. Talukdar, and B. Amrutur, “Spatial reasoning from natural language instructions for robot manipulation,” in IEEE International Conference on Robotics and Automation (ICRA) , 2021
2021
Closest in time.
Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu, “Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs,” in IEEE International Conference on Robotics and Automation (ICRA) , 2021
2021
Closest in time.
W. Yuan, C. Paxton, K. Desingh, and D. Fox, “Sornet: Spatial object-centric representations for sequential manipulation,” 2021
2021
Closest in time.
X. Hong, Y. Lan, L. Pang, J. Guo, and X. Cheng, “Transformation driven visual reasoning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 6903–6912
2021
Closest in time.
S. Chaeibakhsh, R. S. Novin, T. Hermans, A. Merryweather, and A. Kuntz, “Optimizing Hospital Room Layout to Reduce the Risk of Patient Falls,” in International Conference on Operations Research and Enterprise Systems (ICORES) , 2021. [Online]. Available: https://arxiv.org/abs/2101.03210
Original
2021
Closest in time.
M.-H. Guo, J.-X. Cai, Z.-N. Liu, T.-J. Mu, R. R. Martin, and S.-M. Hu, “PCT: Point cloud transformer,” Computational Visual Media , vol. 7, no. 2, pp. 187–199, 2021
2021
Closest in time.
N. Morrical, J. Tremblay, Y. Lin, S. Tyree, S. Birchfield, V. Pascucci, and I. Wald, “Nvisii: A scriptable tool for photorealistic image generation,” arXiv preprint arXiv:2105.13962 , 2021
Original
2021
Closest in time.
A. Qureshi, A. Mousavian, C. Paxton, M. Yip, and D. Fox, “Nerp: Neural rearrangement planning for unknown objects,” in Proceedings of Robotics: Science and Systems , 2021
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…