Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations

Understand

We present Neural Feature Fusion Fields (N3F), a method that improves dense 2D image feature extractors when the latter are applied to the analysis of multiple images reconstructible as a 3D scene.

• Given an image feature extractor, for example pre-trained using self-supervision, N3F uses it as a teacher to learn a student network defined in 3D space.
• The 3D student network is similar to a neural radiance field that distills said features and can be trained with the usual differentiable rendering machinery.
• As a consequence, N3F is readily applicable to most neural rendering formulations, including vanilla NeRF and its extensions to complex dynamic scenes.