Neural Rerendering in the Wild

Understand

We explore total scene capture -- recording, modeling, and rerendering a scene under varying appearance such as season and time of day.

• Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud.
• For each photo, we render the scene points into a deep framebuffer, and train a neural network to learn the mapping of these initial renderings to the actual photos.
• This rerendering network also takes as input a latent appearance vector and a semantic mask indicating the location of transient objects like pedestrians.