Grounded 3D-LLM with Referent Tokens

A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in CVPR , 2017, pp. 5828–5839

2017

Earlier work this paper cites.

B. Graham, M. Engelcke, and L. Van Der Maaten, “3d semantic segmentation with submanifold sparse convolutional networks,” in CVPR , 2018, pp. 9224–9232

2018

Earlier work this paper cites.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

Original

2018

Earlier work this paper cites.

L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia, “Pointgroup: Dual-set point grouping for 3d instance segmentation,” in CVPR , 2020, pp. 4867–4876

2020

Earlier work this paper cites.

D. Z. Chen, A. X. Chang, and M. Nießner, “Scanrefer: 3d object localization in rgb-d scans using natural language,” in ECCV . Springer, 2020, pp. 202–221

2020

Earlier work this paper cites.

P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. Guibas, “Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes,” in ECCV . Springer, 2020, pp. 422–440

2020

Earlier work this paper cites.

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in ECCV . Springer, 2020, pp. 213–229

2020

Earlier work this paper cites.

Z. Liu, Z. Zhang, Y. Cao, H. Hu, and X. Tong, “Group-free 3d object detection via transformers,” in ICCV , 2021, pp. 2949–2958

2021

Earlier work this paper cites.

L. Zhao, D. Cai, L. Sheng, and D. Xu, “3dvg-transformer: Relation modeling for visual grounding on point clouds,” in ICCV , 2021, pp. 2928–2937

2021

Earlier work this paper cites.

D. Z. Chen, Q. Wu, M. Nießner, and A. X. Chang, “D3net: A speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans,” 2021

2021

Earlier work this paper cites.

Z. Chen, A. Gholami, M. Nießner, and A. X. Chang, “Scan2cap: Context-aware dense captioning in rgb-d scans,” in CVPR , 2021, pp. 3193–3203

2021

Earlier work this paper cites.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in ICML . PMLR, 2021, pp. 8748–8763

2021

Earlier work this paper cites.

X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, “Zero-shot detection via vision and language knowledge distillation,” arXiv preprint arXiv:2104.13921 , vol. 2, no. 3, p. 4, 2021

Original

2021

Earlier work this paper cites.

A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang, “Open-vocabulary object detection using captions,” in CVPR , 2021, pp. 14 393–14 402

2021

Earlier work this paper cites.

A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion, “Mdetr-modulated detection for end-to-end multi-modal understanding,” in ICCV , 2021, pp. 1780–1790

2021

Earlier work this paper cites.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

Original

2021

Earlier work this paper cites.

T. Vu, K. Kim, T. M. Luu, T. Nguyen, and C. D. Yoo, “Softgroup for 3d instance segmentation on point clouds,” in CVPR , 2022, pp. 2708–2717

2022

Earlier work this paper cites.

S. Huang, Y. Chen, J. Jia, and L. Wang, “Multi-view transformer for 3d visual grounding,” in CVPR , 2022, pp. 15 524–15 533

2022

Earlier work this paper cites.

S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Language conditioned spatial relation reasoning for 3d object grounding,” NeurIPS , vol. 35, pp. 20 522–20 535, 2022

2022

Earlier work this paper cites.

D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe, “Scanqa: 3d question answering for spatial scene understanding,” in CVPR , 2022, pp. 19 129–19 139

2022

Earlier work this paper cites.

H. Wang, C. Zhang, J. Yu, and W. Cai, “Spatiality-guided transformer for 3D dense captioning on point clouds,” in IJCAI , 2022

2022

Earlier work this paper cites.

D. Cai, L. Zhao, J. Zhang, L. Sheng, and D. Xu, “3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds,” in CVPR , 2022, pp. 16 464–16 473

2022

Earlier work this paper cites.

A. Jain, N. Gkanatsios, I. Mediratta, and K. Fragkiadaki, “Bottom up top down detection transformers for language grounding in images and point clouds,” in ECCV . Springer, 2022, pp. 417–433

2022

Earlier work this paper cites.

L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang et al. , “Grounded language-image pre-training,” in CVPR , 2022, pp. 10 965–10 975

2022

Earlier work this paper cites.

L. Yao, J. Han, Y. Wen, X. Liang, D. Xu, W. Zhang, Z. Li, C. Xu, and H. Xu, “Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection,” NeurIPS , vol. 35, pp. 9125–9138, 2022

2022

Earlier work this paper cites.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” NeurIPS , vol. 35, pp. 23 716–23 736, 2022

2022

Earlier work this paper cites.

S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg et al. , “A generalist agent,” arXiv preprint arXiv:2205.06175 , 2022

Original

2022

Earlier work this paper cites.

X. Ma, S. Yong, Z. Zheng, Q. Li, Y. Liang, S.-C. Zhu, and S. Huang, “Sqa3d: Situated question answering in 3d scenes,” arXiv preprint arXiv:2210.07474 , 2022

Original

2022

Earlier work this paper cites.

Z. Yuan, X. Yan, Y. Liao, Y. Guo, G. Li, S. Cui, and Z. Li, “X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense captioning,” in CVPR , 2022, pp. 8563–8573

2022

Earlier work this paper cites.

Y. Jiao, S. Chen, Z. Jie, J. Chen, L. Ma, and Y.-G. Jiang, “More: Multi-order relation mining for dense captioning in 3d scenes,” in ECCV . Springer, 2022, pp. 528–545

2022

Cited alongside, same era.

X. Yu, L. Tang, Y. Rao, T. Huang, J. Zhou, and J. Lu, “Point-bert: Pre-training 3d point cloud transformers with masked point modeling,” in CVPR , 2022, pp. 19 313–19 322

2022

Cited alongside, same era.

L. Xue, M. Gao, C. Xing, R. Martín-Martín, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, “Ulip: Learning unified representation of language, image and point cloud for 3d understanding,” arXiv preprint arXiv:2212.05171 , 2022

Original

2022

Cited alongside, same era.

B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” in CVPR , 2022, pp. 1290–1299

2022

Cited alongside, same era.

R. Chen, Y. Liu, L. Kong, X. Zhu, Y. Ma, Y. Li, Y. Hou, Y. Qiao, and W. Wang, “Clip2scene: Towards label-efficient 3d scene understanding by clip,” in CVPR , 2023, pp. 7020–7030

2023

Later among the works it cites.

R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi, “Lowis3d: Language-driven open-world instance-level 3d scene understanding,” arXiv preprint arXiv:2308.00353 , 2023

Original

2023

Later among the works it cites.

——, “Pla: Language-driven open-vocabulary 3d scene understanding,” in CVPR , 2023, pp. 7010–7019

2023

Later among the works it cites.

C. Zhu, W. Zhang, T. Wang, X. Liu, and K. Chen, “Object2scene: Putting objects in context for open-vocabulary 3d detection,” arXiv preprint arXiv:2309.09456 , 2023

Original

2023

Later among the works it cites.

S. Peng, K. Genova, C. M. Jiang, A. Tagliasacchi, M. Pollefeys, and T. Funkhouser, “Openscene: 3d scene understanding with open vocabularies,” in CVPR , 2023