Three for one and one for three: Flow, Segmentation, and Surface Normals

Understand

Optical flow, semantic segmentation, and surface normals represent different information modalities, yet together they bring better cues for scene understanding problems.

• In this paper, we study the influence between the three modalities: how one impacts on the others and their efficiency in combination.
• We employ a modular approach using a convolutional refinement network which is trained supervised but isolated from RGB images to enforce joint modality features.
• To assist the training process, we create a large-scale synthetic outdoor dataset that supports dense annotation of semantic segmentation, optical flow, and surface normals.