2019

Large-scale weakly-supervised pre-training for video action recognition

Ghadiyaram, Deepti, Feiszli, Matt, Tran, Du et al.

Understand

Current fully-supervised video datasets consist of only a few hundred thousand videos and fewer than a thousand domain-specific labels.

This hinders the progress towards advanced video architectures.
This paper presents an in-depth study of using large volumes of web videos for pre-training video models for the task of action recognition.
Our primary empirical finding is that pre-training at a very large scale (over 65 million videos), despite on noisy social-media videos and hashtags, substantially improves the state-of-the-art on three challenging public action recognition datasets.

Built on

Non-parametric local transforms for computing visual correspondence
R. Zabih and J. Woodfill · 1994
Earlier work this paper cites.
Space-time interest points
I. Laptev and T. Lindeberg · 2003
Earlier work this paper cites.
Hello! my name is buffy–automatic naming of characters in tv video
M. Everingham, J. Sivic, and A. Zisserman · 2006
Earlier work this paper cites.
Learning realistic human actions from movies
I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld · 2008
Earlier work this paper cites.
Annotating images by mining image search results
X.-J. Wang, L. Zhang, X. Li, and W.-Y. Ma · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
Automatic annotation of human actions in video
O. Duchenne, I. Laptev, J. Sivic, F. Bach, and J. Ponce · 2009
Earlier work this paper cites.
Actions in context
M. Marszalek, I. Laptev, and C. Schmid · 2009
Earlier work this paper cites.
Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach
A. Bergamo and L. Torresani · 2010
Earlier work this paper cites.
Every picture tells a story: Generating sentences from images
A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth · 2010
Earlier work this paper cites.
Learning object categories from internet image searches
R. Fergus, L. Fei-Fei, P. Perona, and A. Zisserman · 2010
Earlier work this paper cites.
HMDB: a large video database for human motion recognition
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre · 2011
Earlier work this paper cites.
Im2text: Describing images using 1 million captioned photographs
V. Ordonez, G. Kulkarni, and T. L. Berg · 2011
Earlier work this paper cites.
Harvesting image databases from the web
F. Schroff, A. Criminisi, and A. Zisserman · 2011
Earlier work this paper cites.
What makes paris look like paris?
C. Doersch, S. Singh, A. Gupta, J. Sivic, and A. Efros · 2012
Earlier work this paper cites.
Learning object class detectors from weakly annotated video
A. Prest, C. Leistner, J. Civera, C. Schmid, and V. Ferrari · 2012
Earlier work this paper cites.
UCF101: A dataset of 101 human actions classes from videos in the wild
K. Soomro, A. R. Zamir, and M. Shah · 2012
Earlier work this paper cites.
Finding actors and actions in movies
P. Bojanowski, F. Bach, I. Laptev, J. Ponce, C. Schmid, and J. Sivic · 2013
Earlier work this paper cites.
Neil: Extracting visual knowledge from web data
X. Chen, A. Shrivastava, and A. Gupta · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean · 2013
Earlier work this paper cites.
Learning everything about anything: Webly-supervised visual concept learning
S. K. Divvala, A. Farhadi, and C. Guestrin · 2014
Earlier work this paper cites.
Decaf: A deep convolutional activation feature for generic visual recognition
J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell · 2014
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
R. Girshick, J. Donahue, T. Darrell, and J. Malik · 2014
Earlier work this paper cites.
THUMOS challenge: Action recognition with a large number of classes
Y. Jiang, J. Liu, A. R. Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar · 2014
Earlier work this paper cites.

Similar

Large-scale video classification with convolutional neural networks
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei · 2014
Cited alongside, same era.
Fully convolutional multi-class multiple instance learning
Original
D. Pathak, E. Shelhamer, J. Long, and T. Darrell · 2014
Cited alongside, same era.
Training deep neural networks on noisy labels with bootstrapping
Original
S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich · 2014
Cited alongside, same era.
Training convolutional networks with noisy labels
Original
S. Sukhbaatar, J. Bruna, M. Paluri, L. Bourdev, and R. Fergus · 2014
Cited alongside, same era.
Hashing for similarity search: A survey
Original
J. Wang, H. T. Shen, J. Song, and J. Ji · 2014
Cited alongside, same era.
The new data and new challenges in multimedia research
Original
B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L. Li · 2015
Cited alongside, same era.
Webly supervised learning of convolutional networks
X. Chen and A. Gupta · 2015
Cited alongside, same era.
Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation
J. Dai, K. He, and J. Sun · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
S. Ioffe and C. Szegedy · 2015
Cited alongside, same era.
Deep classifiers from image tags in the wild
H. Izadinia, B. C. Russell, A. Farhadi, M. D. Hoffman, and A. Hertzmann · 2015
Cited alongside, same era.
Is object localization for free?-weakly-supervised learning with convolutional neural networks
M. Oquab, L. Bottou, I. Laptev, and J. Sivic · 2015
Cited alongside, same era.
Constrained convolutional neural networks for weakly supervised segmentation
D. Pathak, P. Krahenbuhl, and T. Darrell · 2015
Cited alongside, same era.

Then

Quo vadis, action recognition? a new model and the kinetics dataset
J. Carreira and A. Zisserman · 2017
Later among the works it cites.
Wildcat: Weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation
T. Durand, T. Mordan, N. Thome, and M. Cord · 2017
Later among the works it cites.
Accurate, large minibatch sgd: training imagenet in 1 hour
Original
P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He · 2017
Later among the works it cites.
The “something something” video database for learning and evaluating visual common sense
R. Goyal, S. E. Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al · 2017
Later among the works it cites.
Weakly supervised semantic segmentation using web-crawled videos
Original
S. Hong, D. Yeo, S. Kwak, H. Lee, and B. Han · 2017
Later among the works it cites.
Learning visual n-grams from web data
A. Li, A. Jabri, A. Joulin, and L. van der Maaten · 2017
Later among the works it cites.
Learning without forgetting
Z. Li and D. Hoiem · 2017
Later among the works it cites.
Weakly-supervised learning of visual relations
J. Peyre, I. Laptev, C. Schmid, and J. Sivic · 2017
Later among the works it cites.
Transfer learning by ranking for weakly supervised object annotation
Original
Z. Shi, P. Siva, and T. Xiang · 2017
Later among the works it cites.
Revisiting unreasonable effectiveness of data in deep learning era
C. Sun, A. Shrivastava, S. Singh, and A. Gupta · 2017
Later among the works it cites.
The kinetics human action video dataset
Original
W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman · 2017
Later among the works it cites.
Rethinking spatiotemporal feature learning for video understanding
Original
S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy · 2017
Later among the works it cites.
Revisiting the effectiveness of off-the-shelf temporal modeling approaches for large-scale video classification
Original
Y. Bian, C. Gan, X. Liu, F. Li, X. Long, Y. Li, H. Qi, J. Zhou, S. Wen, and Y. Lin · 2017
Later among the works it cites.
Posetrack: A benchmark for human pose estimation and tracking
M. Andriluka, U. Iqbal, A. Milan, E. Insafutdinov, L. Pishchulin, J. Gall, and B. Schiele · 2018
Later among the works it cites.
Object level visual reasoning in videos
Original
F. Baradel, N. Neverova, C. Wolf, J. Mille, and G. Mori · 2018
Later among the works it cites.
A closer look at spatiotemporal convolutions for action recognition
D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri · 2018
Later among the works it cites.
Scaling egocentric vision: The epic-kitchens dataset
D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray · 2018
Later among the works it cites.
Exploring the limits of weakly supervised pretraining
D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten · 2018
Later among the works it cites.
Actionflownet: Learning motion representation for action recognition
J. Y.-H. Ng, J. Choi, J. Neumann, and L. S. Davis · 2018
Later among the works it cites.
Non-local neural networks
X. Wang, R. Girshick, A. Gupta, and K. He · 2018
Later among the works it cites.
Videos as space-time region graphs
Original
X. Wang and A. Gupta · 2018
Later among the works it cites.
ECO: efficient convolutional network for online video understanding
Original
M. Zolfaghari, K. S. Singh, and T. Brox · 2018
Later among the works it cites.
SCSampler: Sampling salient clips from video for efficient action recognition
Original
B. Korbar, D. Tran, L. Torresani · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…