2024

FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding

Zuo, Xingxing, Samangouei, Pouya, Zhou, Yunwen et al.

Understand

Precisely perceiving the geometric and semantic properties of real-world 3D objects is crucial for the continued evolution of augmented reality and robotic applications.

To this end, we present Foundation Model Embedded Gaussian Splatting (FMGS), which incorporates vision-language embeddings of foundation models into 3D Gaussian Splatting (GS).
The key contribution of this work is an efficient method to reconstruct and represent 3D vision-language models.
This is achieved by distilling feature maps generated from image-based foundation models into those rendered from our 3D model.

Built on

Ewa volume splatting
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross · 2001
Earlier work this paper cites.
Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera
Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al · 2011
Earlier work this paper cites.
Structure-from-motion revisited
Johannes L Schonberger and Jan-Michael Frahm · 2016
Earlier work this paper cites.
Structure-from-motion revisited
Johannes L Schonberger and Jan-Michael Frahm · 2016
Earlier work this paper cites.
Scannet: Richly-annotated 3d reconstructions of indoor scenes
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner · 2017
Earlier work this paper cites.
Pointnet: Deep learning on point sets for 3d classification and segmentation
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas · 2017
Earlier work this paper cites.
Iqa: Visual question answering in interactive environments
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi · 2018
Earlier work this paper cites.
Volumetric instance-aware semantic mapping and 3d object discovery
Margarita Grinvald, Fadri Furrer, Tonci Novkovic, Jen Jen Chung, Cesar Cadena, Roland Siegwart, and Juan Nieto · 2019
Earlier work this paper cites.
Panopticfusion: Online volumetric semantic mapping at the level of stuff and things
Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji · 2019
Earlier work this paper cites.
Surfelmeshing: Online surfel-based mesh reconstruction
Thomas Schöps, Torsten Sattler, and Marc Pollefeys · 2019
Earlier work this paper cites.
Differentiable surface splatting for point-based geometry processing
Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung · 2019
Earlier work this paper cites.
Nerf: Representing scenes as neural radiance fields for view synthesis
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng · 2020
Earlier work this paper cites.
Mobile3drecon: real-time monocular 3d reconstruction on a mobile phone
Xingbin Yang, Liyang Zhou, Hanqing Jiang, Zhongliang Tang, Yuanbo Wang, Hujun Bao, and Guofeng Zhang · 2020
Earlier work this paper cites.
Deep vit features as dense visual descriptors
Original
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel · 2021
Earlier work this paper cites.
Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan · 2021
Earlier work this paper cites.
Emerging properties in self-supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin · 2021
Earlier work this paper cites.
Open-vocabulary image segmentation
Original
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin · 2021
Earlier work this paper cites.
Open-vocabulary object detection via vision and language knowledge distillation
Original
Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui · 2021
Earlier work this paper cites.
Differentiable slam-net: Learning particle slam for visual navigation
Peter Karkus, Shaojun Cai, and David Hsu · 2021
Earlier work this paper cites.
End-to-end human pose and mesh reconstruction with transformers
Kevin Lin, Lijuan Wang, and Zicheng Liu · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Earlier work this paper cites.
Neuralrecon: Real-time coherent 3d reconstruction from monocular video
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao · 2021
Earlier work this paper cites.

Similar

Pointclip: Point cloud understanding by clip
Original
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li · 2021
Cited alongside, same era.
Flamingo: a visual language model for few-shot learning
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al · 2022
Cited alongside, same era.
Scanqa: 3d question answering for spatial scene understanding
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe · 2022
Cited alongside, same era.
Mip-nerf 360: Unbounded anti-aliased neural radiance fields
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman · 2022
Cited alongside, same era.
Simvqa: Exploring simulated environments for visual question answering
Paola Cascante-Bonilla, Hui Wu, Letao Wang, Rogerio S Feris, and Vicente Ordonez · 2022
Cited alongside, same era.
Tensorf: Tensorial radiance fields
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su · 2022
Cited alongside, same era.
Reproducible scaling laws for contrastive language-image learning
Original
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev · 2022
Cited alongside, same era.
Voxel-informed language grounding
Original
Rodolfo Corona, Shizhan Zhu, Dan Klein, and Trevor Darrell · 2022
Cited alongside, same era.
Decomposing nerf for editing via feature field distillation
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann · 2022
Cited alongside, same era.
Language-driven semantic segmentation
Original
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl · 2022
Cited alongside, same era.
Open-vocabulary semantic segmentation with mask-adapted clip
Original
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu · 2022
Cited alongside, same era.
Image segmentation using text and image prompts
Timo Lüddecke and Alexander Ecker · 2022
Cited alongside, same era.

Then

Segment anything in 3d with nerfs
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Chen Yang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian · 2023
Later among the works it cites.
Panoptic vision-language feature fields
Original
Haoran Chen, Kenneth Blomqvist, Francesco Milano, and Roland Siegwart · 2023
Later among the works it cites.
Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning
Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio Torralba, Florian Shkurti, and Liam Paull · 2023
Later among the works it cites.
Conceptfusion: Open-set multimodal 3d mapping, 2023
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba · 2023
Later among the works it cites.
Splatam: Splat, track & map 3d gaussians for dense rgb-d slam
Original
Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten · 2023
Later among the works it cites.
3d gaussian splatting for real-time radiance field rendering
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis · 2023
Later among the works it cites.
Lerf: Language embedded radiance fields
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik · 2023
Later among the works it cites.
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al · 2023
Later among the works it cites.
Weakly supervised 3d open-vocabulary segmentation
Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu · 2023
Later among the works it cites.
3d open-vocabulary segmentation with foundation models
Original
Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu · 2023
Later among the works it cites.
OVIR-3d: Open-vocabulary 3d instance retrieval without training on 3d data
Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boularias, and Kostas Bekris · 2023
Later among the works it cites.
Open-vocabulary point-cloud object detection without 3d annotation
Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang · 2023
Later among the works it cites.
OpenMask3D: Open-Vocabulary 3D Instance Segmentation
Ayça Takmaz, Elisabetta Fedele, Robert W. Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann · 2023
Later among the works it cites.
Vl-fields: Towards language-grounded neural implicit spatial representations
Original
Nikolaos Tsagkas, Oisin Mac Aodha, and Chris Xiaoxuan Lu · 2023
Later among the works it cites.
Ulip-2: Towards scalable multimodal pre-training for 3d understanding, 2023
Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese · 2023
Later among the works it cites.
Featurenerf: Learning generalizable nerfs by distilling pre-trained vision foundation models
Original
Jianglong Ye, Naiyan Wang, and Xiaolong Wang · 2023
Later among the works it cites.
Multi-task real robot learning with generalizable neural feature fields
Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuying Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xiaolong Wang · 2023
Later among the works it cites.
A survey on 3d gaussian splatting
Original
Guikun Chen and Wenguan Wang · 2024
Closest in time.
3d gaussian as a new vision era: A survey
Original
Ben Fei, Jingyi Xu, Rui Zhang, Qingyuan Zhou, Weidong Yang, and Ying He · 2024
Closest in time.
Langsplat: 3d language gaussian splatting
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister · 2024
Closest in time.
LEGaussians: Language embedded 3d gaussians for open-vocabulary scene understanding
Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan · 2024
Closest in time.
Feature 3DGS: Supercharging 3d gaussian splatting to enable distilled feature fields
Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…