Pose-conditioned Spatio-Temporal Attention for Human Action Recognition

Understand

We address human action recognition from multi-modal video data involving articulated pose and RGB frames and propose a two-stream approach.

• The pose stream is processed with a convolutional model taking as input a 3D tensor holding data from a sub-sequence.
• A specific joint ordering, which respects the topology of the human body, ensures that different convolutional layers correspond to meaningful levels of abstraction.
• The raw RGB stream is handled by a spatio-temporal soft-attention mechanism conditioned on features from the pose network.