Multiview Supervision By Registration

Understand

This paper presents a semi-supervised learning framework to train a keypoint detector using multiview image streams given the limited labeled data (typically $<$4\%).

• We leverage the complementary relationship between multiview geometry and visual tracking to provide three types of supervisionary signals to utilize the unlabeled data: (1) keypoint detection in one view can be supervised by other views via the epipolar geometry; (2) a keypoint moves smoothly over time where its optical flow can be used to temporally supervise consecutive image frames to each other; (3) visible keypoint in one view is likely to be visible in the adjacent view.
• We integrate these three signals in a differentiable fashion to design a new end-to-end neural network composed of three pathways.
• This design allows us to extensively use the unlabeled data to train the keypoint detector.