Learning to Localize and Align Fine-Grained Actions to Sparse Instructions

Understand

Automatic generation of textual video descriptions that are time-aligned with video content is a long-standing goal in computer vision.

• The task is challenging due to the difficulty of bridging the semantic gap between the visual and natural language domains.
• This paper addresses the task of automatically generating an alignment between a set of instructions and a first person video demonstrating an activity.
• The sparse descriptions and ambiguity of written instructions create significant alignment challenges.