2023

ASSIST: Interactive Scene Nodes for Scalable and Realistic Indoor Simulation

Zhong, Zhide, Cao, Jiakai, Gu, Songen et al.

Understand

We present ASSIST, an object-wise neural radiance field as a panoptic representation for compositional and realistic simulation.

Central to our approach is a novel scene node data structure that stores the information of each object in a unified fashion, allowing online interaction in both intra- and cross-scene settings.
By incorporating a differentiable neural network along with the associated bounding box and semantic features, the proposed structure guarantees user-friendly interaction on independent objects to scale up novel view simulation.
Objects in the scene can be queried, added, duplicated, deleted, transformed, or swapped simply through mouse/keyboard controls or language instructions.

Built on

J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 4104–4113
2016
Earlier work this paper cites.
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 5828–5839
2017
Earlier work this paper cites.
Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, “Habitat: A platform for embodied AI research,” in Intl. Conf. on Computer Vision (ICCV) , 2019
2019
Earlier work this paper cites.
M. Deitke, W. Han, A. Herrasti, A. Kembhavi, E. Kolve, R. Mottaghi, J. Salvador, D. Schwenk, E. VanderBilt, M. Wallingford, L. Weihs, M. Yatskar, and A. Farhadi, “RoboTHOR: An open simulation-to-real embodied AI platform,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2020
2020
Earlier work this paper cites.
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in European Conf. on Computer Vision (ECCV) . Springer, 2020, pp. 405–421
2020
Earlier work this paper cites.
J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields,” in Intl. Conf. on Computer Vision (ICCV) , 2021, pp. 5835–5844
2021
Earlier work this paper cites.
J. Ost, F. Mannan, N. Thuerey, J. Knodt, and F. Heide, “Neural scene graphs for dynamic scenes,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2021, pp. 2856–2865
2021
Earlier work this paper cites.
B. Yang, Y. Zhang, Y. Xu, Y. Li, H. Zhou, H. Bao, G. Zhang, and Z. Cui, “Learning object-compositional neural radiance field for editable scene rendering,” in Intl. Conf. on Computer Vision (ICCV) , 2021, pp. 13 779–13 788
2021
Earlier work this paper cites.
S. Zhi, T. Laidlow, S. Leutenegger, and A. J. Davison, “In-place scene labelling and understanding with implicit scene representation,” in Intl. Conf. on Computer Vision (ICCV) , 2021, pp. 15 838–15 847
2021
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in Intl. Conf. on Machine Learning (ICML) . PMLR, 2021, pp. 8748–8763
2021
Earlier work this paper cites.
A. Eftekhar, A. Sax, J. Malik, and A. Zamir, “Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans,” in Intl. Conf. on Computer Vision (ICCV) , 2021, pp. 10 786–10 796
2021
Earlier work this paper cites.
A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817 , 2022
Original
2022
Earlier work this paper cites.
C. Li, F. Xia, R. Martín-Martín, M. Lingelbach, S. Srivastava, B. Shen, K. E. Vainio, C. Gokmen, G. Dharan, T. Jain, A. Kurenkov, K. Liu, H. Gweon, J. Wu, L. Fei-Fei, and S. Savarese, “igibson 2.0: Object-centric simulation for robot learning of everyday household tasks,” in Conf. on Robot Learning , vol. 164, 2022, pp. 455–465
2022
Earlier work this paper cites.

Similar

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-NeRF 360: Unbounded anti-liased neural radiance fields,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2022
2022
Cited alongside, same era.
A. Kundu, K. Genova, X. Yin, A. Fathi, C. Pantofaru, L. J. Guibas, A. Tagliasacchi, F. Dellaert, and T. Funkhouser, “Panoptic neural fields: A semantic object-aware neural scene representation,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 12 871–12 881
2022
Cited alongside, same era.
Q. Wu, X. Liu, Y. Chen, K. Li, C. Zheng, J. Cai, and J. Zheng, “Object-compositional neural implicit surfaces,” in European Conf. on Computer Vision (ECCV) . Springer, 2022, pp. 197–213
2022
Cited alongside, same era.
R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky, “Resolution-robust large mask inpainting with fourier convolutions,” in IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV) , 2022, pp. 2149–2159
2022
Cited alongside, same era.
Z. Yu, S. Peng, M. Niemeyer, T. Sattler, and A. Geiger, “Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction,” Advances in Neural Information Processing Systems (NIPS) , vol. 35, pp. 25 018–25 032, 2022
2022
Cited alongside, same era.
H. K. Cheng and A. G. Schwing, “Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,” in European Conf. on Computer Vision (ECCV) . Springer, 2022, pp. 640–658
2022
Cited alongside, same era.
D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine, “ViNT: A foundation model for visual navigation,” arXiv pre-print , 2023. [Online]. Available: https://arxiv.org/abs/2306.14846
Original
2023
Cited alongside, same era.
Z. Yang, Y. Chen, J. Wang, S. Manivasagam, W.-C. Ma, A. J. Yang, and R. Urtasun, “Unisim: A neural closed-loop sensor simulator,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2023, pp. 1389–1399
2023
Cited alongside, same era.
A. Byravan, J. Humplik, L. Hasenclever, A. Brussee, F. Nori, T. Haarnoja, B. Moran, S. Bohez, F. Sadeghi, B. Vujatovic et al. , “Nerf2real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields,” in IEEE Intl. Conf. on Robotics and Automation (ICRA) . IEEE, 2023, pp. 9362–9369
2023
Cited alongside, same era.
T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter et al. , “Scaling robot learning with semantically imagined experience,” in Robotics: Science and Systems (RSS) , 2023
2023
Cited alongside, same era.
A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn et al. , “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818 , 2023
Original
2023
Cited alongside, same era.
Z. Chen, S. Kiami, A. Gupta, and V. Kumar, “Genaug: Retargeting behaviors to unseen situations via generative augmentation,” arXiv preprint arXiv:2302.06671 , 2023
Original
2023
Cited alongside, same era.

Then

I. Kapelyukh, V. Vosylius, and E. Johns, “Dall-e-bot: Introducing web-scale diffusion models to robotics,” IEEE Robotics and Automation Letters , 2023
2023
Closest in time.
Z. Wu, T. Liu, L. Luo, Z. Zhong, J. Chen, H. Xiao, C. Hou, H. Lou, Y. Chen, R. Yang et al. , “Mars: An instance-aware, modular and realistic simulator for autonomous driving,” arXiv preprint arXiv:2307.15058 , 2023
Original
2023
Closest in time.
X. Kong, S. Liu, M. Taher, and A. J. Davison, “vmap: Vectorised object mapping for neural field slam,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2023, pp. 952–961
2023
Closest in time.
X. Han, H. Liu, Y. Ding, and L. Yang, “Ro-map: Real-time multi-object mapping with neural radiance fields,” IEEE Robotics and Automation Letters , 2023
2023
Closest in time.
W. Bing, L. Chen, and B. Yang, “Dm-nerf: 3d scene geometry decomposition and manipulation from 2d images,” in Intl. Conf. on Learning Representations (ICLR) , 2023
2023
Closest in time.
Y. Wang, W. Wu, and D. Xu, “Learning unified decompositional and compositional nerf for editable novel view synthesis,” in Intl. Conf. on Computer Vision (ICCV) , 2023
2023
Closest in time.
Y. Siddiqui, L. Porzi, S. R. Bulò, N. Müller, M. Nießner, A. Dai, and P. Kontschieder, “Panoptic lifting for 3d scene understanding with neural fields,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2023, pp. 9043–9052
2023
Closest in time.
A. Haque, M. Tancik, A. Efros, A. Holynski, and A. Kanazawa, “Instruct-nerf2nerf: Editing 3d scenes with instructions,” in Intl. Conf. on Computer Vision (ICCV) , 2023
2023
Closest in time.
X. Wang, J. Zhu, Q. Ye, Y. Huo, Y. Ran, Z. Zhong, and J. Chen, “Seal-3d: Interactive pixel-level editing for neural radiance fields,” in Intl. Conf. on Computer Vision (ICCV) , 2023
2023
Closest in time.
C. Jambon, B. Kerbl, G. Kopanas, S. Diolatzis, G. Drettakis, and T. Leimkühler, “Nerfshop: Interactive editing of neural radiance fields,” Proc. of the ACM on Computer Graphics and Interactive Techniques , vol. 6, no. 1, 2023
2023
Closest in time.
R. OpenAI, “Gpt-4 technical report,” arXiv , p. 2303:08774, 2023
2023
Closest in time.
M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja et al. , “Nerfstudio: A modular framework for neural radiance field development,” in SIGGRAPH , 2023, pp. 1–12
Original
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…