Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition

Understand

We propose a self-supervised learning method to jointly reason about spatial and temporal context for video recognition.

• Recent self-supervised approaches have used spatial context [9, 34] as well as temporal coherency [32] but a combination of the two requires extensive preprocessing such as tracking objects through millions of video frames [59] or computing optical flow to determine frame regions with high motion [30].
• We propose to combine spatial and temporal context in one self-supervised framework without any heavy preprocessing.
• We divide multiple video frames into grids of patches and train a network to solve jigsaw puzzles on these patches from multiple frames.