Unsupervised Learning from Narrated Instruction Videos

Understand

We address the problem of automatically learning the main steps to complete a certain task, such as changing a car tire, from a set of narrated instruction videos.

• The contributions of this paper are three-fold.
• First, we develop a new unsupervised learning approach that takes advantage of the complementary nature of the input video and the associated narration.
• The method solves two clustering problems, one in text and one in video, applied one after each other and linked by joint constraints to obtain a single coherent sequence of steps in both modalities.