Spatio-Temporal Crop Aggregation for Video Representation Learning

Understand

We propose Spatio-temporal Crop Aggregation for video representation LEarning (SCALE), a novel method that enjoys high scalability at both training and inference time.

• Our model builds long-range video features by learning from sets of video clip-level features extracted with a pre-trained backbone.
• To train the model, we propose a self-supervised objective consisting of masked clip feature prediction.
• We apply sparsity to both the input, by extracting a random set of video clips, and to the loss function, by only reconstructing the sparse inputs.