2020

Audiovisual SlowFast Networks for Video Recognition

Xiao, Fanyi, Lee, Yong Jae, Grauman, Kristen et al.

Understand

We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception.

AVSlowFast has Slow and Fast visual pathways that are deeply integrated with a Faster Audio pathway to model vision and sound in a unified representation.
We fuse audio and visual features at multiple layers, enabling audio to contribute to the formation of hierarchical audiovisual concepts.
To overcome training difficulties that arise from different learning dynamics for audio and visual modalities, we introduce DropPathway, which randomly drops the Audio pathway during training as an effective regularization technique.

Built on

Hearing lips and seeing voices
Harry McGurk and John MacDonald · 1976
Earlier work this paper cites.
The statistics of natural images
Daniel L Ruderman · 1994
Earlier work this paper cites.
Statistics of natural images and models
Jinggang Huang and David Mumford · 1999
Earlier work this paper cites.
Tests of auditory–visual integration efficiency within the framework of the fuzzy logical model of perception
Dominic W Massaro and Michael M Cohen · 2000
Earlier work this paper cites.
Audio-visual scene analysis: evidence for a ”very-early” integration process in audio-visual speech perception
Jean-Luc Schwartz, Frederic Berthommier, and Christophe Savariaux · 2002
Earlier work this paper cites.
Audiovisual mirror neurons and action recognition
Christian Keysers, Evelyne Kohler, M Alessandra Umiltà, Luca Nanetti, Leonardo Fogassi, and Vittorio Gallese · 2003
Earlier work this paper cites.
Recent advances in the automatic recognition of audiovisual speech
Gerasimos Potamianos, Chalapathy Neti, Guillaume Gravier, Ashutosh Garg, and Andrew W Senior · 2003
Earlier work this paper cites.
Fusion and combination in audio-visual integration
Kei Omata and Ken Mogi · 2007
Earlier work this paper cites.
Blind audiovisual source separation based on sparse redundant representations
Anna Llagostera Casanovas, Gianluca Monaci, Pierre Vandergheynst, and Remi Gribonval · 2010
Earlier work this paper cites.
HMDB: a large video database for human motion recognition
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre · 2011
Earlier work this paper cites.
Multimodal deep learning
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng · 2011
Earlier work this paper cites.
Improving neural networks by preventing co-adaptation of feature detectors
Original
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov · 2012
Earlier work this paper cites.
A dataset of 101 human action classes from videos in the wild
Original
Khurram Soomro, Amir Roshan Zamir, and M Shah · 2012
Earlier work this paper cites.
The new handbook of multisensory processing
Barry E Stein · 2012
Earlier work this paper cites.
Auditory perceptual learning for speech perception can be enhanced by audiovisual training
Lynne Bernstein, Edward Auer, Jintao Jiang, and Silvio Eberhardt · 2013
Earlier work this paper cites.
Multimodal information improves the rapid detection of mental fatigue
François Laurent, Mario Valderrama, Michel Besserve, Mathias Guillard, Jean-Philippe Lachaux, Jacques Martinerie, and Geneviève Florence · 2013
Earlier work this paper cites.
Return of the devil in the details: Delving deep into convolutional nets
K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman · 2014
Earlier work this paper cites.
Two-Stream Convolutional Networks for Action Recognition in Videos
K. Simonyan and A. Zisserman · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Faster R-CNN: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun · 2015
Earlier work this paper cites.
Very Deep Convolutional Networks for Large-Scale Image Recognition
K. Simonyan and A. Zisserman · 2015
Earlier work this paper cites.
Learning spatiotemporal features with 3d convolutional networks
D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri · 2015
Earlier work this paper cites.
Soundnet: Learning sound representations from unlabeled video
Yusuf Aytar, Carl Vondrick, and Antonio Torralba · 2016
Earlier work this paper cites.
Out of time: automated lip sync in the wild
Joon Son Chung and Andrew Zisserman · 2016
Earlier work this paper cites.
Spatiotemporal residual networks for video action recognition
Christoph Feichtenhofer, Axel Pinz, and Richard Wildes · 2016
Earlier work this paper cites.
Convolutional two-stream network fusion for video action recognition
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman · 2016
Earlier work this paper cites.
Multimodal compact bilinear pooling for visual question answering and visual grounding
Original
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach · 2016
Earlier work this paper cites.
SGDR: Stochastic gradient descent with warm restarts
Original
Ilya Loshchilov and Frank Hutter · 2016
Earlier work this paper cites.
Shuffle and learn: unsupervised learning using temporal order verification
Ishan Misra, C Lawrence Zitnick, and Martial Hebert · 2016
Earlier work this paper cites.
Ambient sound provides supervision for visual learning
Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba · 2016
Earlier work this paper cites.

Similar

Hollywood in homes: Crowdsourcing data collection for activity understanding
Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta · 2016
Cited alongside, same era.
Temporal segment networks: Towards good practices for deep action recognition
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool · 2016
Cited alongside, same era.
Look, listen and learn
Relja Arandjelovic and Andrew Zisserman · 2017
Cited alongside, same era.
Gated multimodal units for information fusion
Original
John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González · 2017
Cited alongside, same era.
Revisiting the effectiveness of off-the-shelf temporal modeling approaches for large-scale video classification
Original
Yunlong Bian, Chuang Gan, Xiao Liu, Fu Li, Xiang Long, Yandong Li, Heng Qi, Jie Zhou, Shilei Wen, and Yuanqing Lin · 2017
Cited alongside, same era.
Quo vadis, action recognition? a new model and the kinetics dataset
Joao Carreira and Andrew Zisserman · 2017
Cited alongside, same era.
Lip reading sentences in the wild
Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman · 2017
Cited alongside, same era.
Self-supervised video representation learning with odd-one-out networks
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould · 2017
Cited alongside, same era.
Learning to fly by crashing
Dhiraj Gandhi, Lerrel Pinto, and Abhinav Gupta · 2017
Cited alongside, same era.
Accurate, large minibatch SGD: Training ImageNet in 1 hour
Original
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He · 2017
Cited alongside, same era.
Mask R-CNN
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick · 2017
Cited alongside, same era.
CNN architectures for large-scale audio classification
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al · 2017
Cited alongside, same era.

Then

Attention clusters: Purely attention based local feature integration for video classification
Xiang Long, Chuang Gan, Gerard De Melo, Jiajun Wu, Xiao Liu, and Shilei Wen · 2018
Later among the works it cites.
Seeing voices and hearing faces: Cross-modal biometric matching
Arsha Nagrani, Samuel Albanie, and Andrew Zisserman · 2018
Later among the works it cites.
Audio-visual scene analysis with self-supervised multisensory features
Andrew Owens and Alexei A Efros · 2018
Later among the works it cites.
Learning to localize sound source in visual scenes
Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon · 2018
Later among the works it cites.
Actor-centric relation network
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid · 2018
Later among the works it cites.
A closer look at spatiotemporal convolutions for action recognition
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri · 2018
Later among the works it cites.
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He · 2018
Later among the works it cites.
Videos as space-time region graphs
Xiaolong Wang and Abhinav Gupta · 2018
Later among the works it cites.
Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy · 2018
Later among the works it cites.
Pixel objectness: Learning to segment generic objects automatically in images and videos
B. Xiong, S. Jain, and K. Grauman · 2018
Later among the works it cites.
The sound of pixels
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba · 2018
Later among the works it cites.
ECO: Efficient convolutional network for online video understanding
Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox · 2018
Later among the works it cites.
SlowFast Networks for Video Recognition
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He · 2019
Later among the works it cites.
What would you expect? anticipating egocentric actions with rolling-unrolling LSTMs and modality attention
Antonino Furnari and Giovanni Maria Farinella · 2019
Later among the works it cites.
Video Action Transformer Network
Rohit Girdhar, João Carreira, Carl Doersch, and Andrew Zisserman · 2019
Later among the works it cites.
Video representation learning by dense predictive coding
Tengda Han, Weidi Xie, and Andrew Zisserman · 2019
Later among the works it cites.
Momentum contrast for unsupervised visual representation learning
Original
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick · 2019
Later among the works it cites.
Deep multimodal clustering for unsupervised audiovisual learning
Di Hu, Feiping Nie, and Xuelong Li · 2019
Later among the works it cites.
Timeception for complex action recognition
Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders · 2019
Later among the works it cites.
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen · 2019
Later among the works it cites.
Self-supervised video representation learning with space-time cubic puzzles
Dahun Kim, Donghyeon Cho, and In So Kweon · 2019
Later among the works it cites.
Revisiting self-supervised visual representation learning
Original
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer · 2019
Later among the works it cites.
SCSampler: Sampling salient clips from video for efficient action recognition
Bruno Korbar, Du Tran, and Lorenzo Torresani · 2019
Later among the works it cites.
Multisensory Interactions in the Real World
Salvador Soto-Faraco, Daria Kvasova, Emmanuel Biau, Nara Ikumi, Manuela Ruzzoli, Luis Morís-Fernández, and Mireia Torralba · 2019
Later among the works it cites.
FBK-HUPBA Submission to the EPIC-Kitchens 2019 Action Recognition Challenge
Original
Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz · 2019
Later among the works it cites.
Hierarchical feature aggregation networks for video action recognition
Original
Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz · 2019
Later among the works it cites.
Contrastive bidirectional transformer for temporal representation learning
Original
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid · 2019
Later among the works it cites.
Video classification with channel-separated convolutional networks
Original
Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli · 2019
Later among the works it cites.
Long-term feature banks for detailed video understanding
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…