Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph

A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 5828–5839

2017

Earlier work this paper cites.

I. Armeni, Z.-Y. He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese, “3d scene graph: A structure for unified semantics, 3d space, and camera,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 5664–5673

2019

Earlier work this paper cites.

U.-H. Kim, J.-M. Park, T.-J. Song, and J.-H. Kim, “3-d scene graph: A sparse and semantic representation of physical environments for intelligent agents,” IEEE transactions on cybernetics , vol. 50, no. 12, pp. 4921–4933, 2019

2019

Earlier work this paper cites.

J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, et al. , “The replica dataset: A digital replica of indoor spaces,” arXiv preprint arXiv:1906.05797 , 2019

Original

2019

Earlier work this paper cites.

J. Wald, H. Dhamo, N. Navab, and F. Tombari, “Learning 3d semantic scene graphs from 3d indoor reconstructions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 3961–3970

2020

Earlier work this paper cites.

D. Z. Chen, A. X. Chang, and M. Nießner, “Scanrefer: 3d object localization in rgb-d scans using natural language,” in European conference on computer vision . Springer, 2020, pp. 202–221

2020

Earlier work this paper cites.

P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. J. Guibas, “ReferIt3D: Neural listeners for fine-grained 3d object identification in real-world scenes,” in 16th European Conference on Computer Vision (ECCV) , 2020

2020

Earlier work this paper cites.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

2021

Earlier work this paper cites.

M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill, “Multimodal few-shot learning with frozen language models,” Advances in Neural Information Processing Systems , vol. 34, pp. 200–212, 2021

2021

Earlier work this paper cites.

S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel, “Deep vit features as dense visual descriptors,” arXiv preprint arXiv:2112.05814 , vol. 2, no. 3, p. 4, 2021

Original

2021

Earlier work this paper cites.

L. Zhao, D. Cai, L. Sheng, and D. Xu, “3dvg-transformer: Relation modeling for visual grounding on point clouds,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 2928–2937

2021

Earlier work this paper cites.

C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” IEEE Transactions on Robotics , vol. 37, no. 6, pp. 1874–1890, 2021

2021

Earlier work this paper cites.

N. Mu, A. Kirillov, D. Wagner, and S. Xie, “Slip: Self-supervision meets language-image pre-training,” in European conference on computer vision . Springer, 2022, pp. 529–544

2022

Earlier work this paper cites.

W. Chen, S. Hu, R. Talak, and L. Carlone, “Leveraging large language models for robot 3d scene understanding,” arXiv preprint arXiv:2209.05629 , 2022

Original

2022

Earlier work this paper cites.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

2022

Earlier work this paper cites.

H. Ha and S. Song, “Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models,” arXiv preprint arXiv:2207.11514 , 2022

Original

2022

Earlier work this paper cites.

N. Hughes, Y. Chang, and L. Carlone, “Hydra: A real-time spatial perception system for 3d scene graph construction and optimization,” arXiv preprint arXiv:2201.13360 , 2022

Original

2022

Earlier work this paper cites.

J. Wald, N. Navab, and F. Tombari, “Learning 3d semantic scene graphs with instance embeddings,” International Journal of Computer Vision , vol. 130, no. 3, pp. 630–651, 2022

2022

Earlier work this paper cites.

S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Language conditioned spatial relation reasoning for 3d object grounding,” Advances in neural information processing systems , vol. 35, pp. 20 522–20 535, 2022

2022

Earlier work this paper cites.

J. Li, P. Wang, P. Xiong, T. Cai, Z. Yan, L. Yang, J. Liu, H. Fan, and S. Liu, “Practical stereo matching via cascaded recurrent network with adaptive correlation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 16 263–16 272

2022

Earlier work this paper cites.

Y. Fang, Q. Sun, X. Wang, T. Huang, X. Wang, and Y. Cao, “Eva-02: A visual representation for neon genesis,” arXiv preprint arXiv:2303.11331 , 2023

Original

2023

Earlier work this paper cites.

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 11 975–11 986

2023

Earlier work this paper cites.

Z. Sun, Y. Fang, T. Wu, P. Zhang, Y. Zang, S. Kong, Y. Xiong, D. Lin, and J. Wang, “Alpha-clip: A clip model focusing on wherever you want,” arXiv preprint arXiv:2312.03818 , 2023

Original

2023

Earlier work this paper cites.

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. , “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4015–4026

2023

Earlier work this paper cites.

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” 2023

2023

Earlier work this paper cites.

R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 15 180–15 190

2023

Earlier work this paper cites.

J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

2023

Earlier work this paper cites.

M. Ranzinger, G. Heinrich, J. Kautz, and P. Molchanov, “Am-radio: Agglomerative model–reduce all domains into one,” arXiv preprint arXiv:2312.06709 , 2023

Original

2023

Earlier work this paper cites.

S.-C. Wu, K. Tateno, N. Navab, and F. Tombari, “Incremental 3d semantic scene graph prediction from rgb sequences,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 5064–5074

2023

Earlier work this paper cites.

H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” 2023

2023

Cited alongside, same era.

K. Kim, K. Yoon, J. Jeon, Y. In, J. Moon, D. Kim, and C. Park, “Llm4sgg: Large language model for weakly supervised scene graph generation,” arXiv e-prints , pp. arXiv–2310, 2023

2023

Cited alongside, same era.

Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan, “3d-llm: Injecting the 3d world into large language models,” Advances in Neural Information Processing Systems , vol. 36, pp. 20 482–20 494, 2023

2023

Cited alongside, same era.

H. Huang, Z. Wang, R. Huang, L. Liu, X. Cheng, Y. Zhao, T. Jin, and Z. Zhao, “Chat-3d v2: Bridging 3d scene and large language models with object identifiers,” arXiv preprint arXiv:2312.08168 , 2023

Original

2023

Cited alongside, same era.

Q. Sun, J. Wang, Q. Yu, Y. Cui, F. Zhang, X. Zhang, and X. Wang, “Eva-clip-18b: Scaling clip to 18 billion parameters,” arXiv preprint arXiv:2402.04252 , 2024

Original

2024

Closest in time.

X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J. Gao, and Y. J. Lee, “Segment everything everywhere all at once,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024. [Online]. Available: https://llava-vl.github.io/blog/2024-01-30-llava-next/

2024

Closest in time.

S. Koch, N. Vaskevicius, M. Colosi, P. Hermosilla, and T. Ropinski, “Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships,” arXiv preprint arXiv:2402.12259 , 2024

Original

2024

Closest in time.