Action Classification and Highlighting in Videos

Understand

Inspired by recent advances in neural machine translation, that jointly align and translate using encoder-decoder networks equipped with attention, we propose an attentionbased LSTM model for human activity recognition.

• Our model jointly learns to classify actions and highlight frames associated with the action, by attending to salient visual information through a jointly learned soft-attention networks.
• We explore attention informed by various forms of visual semantic features, including those encoding actions, objects and scenes.
• We qualitatively show that soft-attention can learn to effectively attend to important objects and scene information correlated with specific human actions.