Fast Weakly Supervised Action Segmentation Using Mutual Consistency

Understand

Action segmentation is the task of predicting the actions for each frame of a video.

• As obtaining the full annotation of videos for action segmentation is expensive, weakly supervised approaches that can learn only from transcripts are appealing.
• In this paper, we propose a novel end-to-end approach for weakly supervised action segmentation based on a two-branch neural network.
• The two branches of our network predict two redundant but different representations for action segmentation and we propose a novel mutual consistency (MuCon) loss that enforces the consistency of the two redundant representations.