2021

MoViNets: Mobile Video Networks for Efficient Video Recognition

Kondratyuk, Dan, Yuan, Liangzhe, Li, Yandong et al.

Understand

We present Mobile Video Networks (MoViNets), a family of computation and memory efficient video networks that can operate on streaming video for online inference.

3D convolutional neural networks (CNNs) are accurate at video recognition but require large computation and memory budgets and do not support online inference, making them difficult to work on mobile devices.
We propose a three-step approach to improve computational efficiency while substantially reducing the peak memory usage of 3D CNNs.
First, we design a video network search space and employ neural architecture search to generate efficient and diverse 3D CNN architectures.

Built on

Acceleration of stochastic approximation by averaging
Boris T Polyak and Anatoli B Juditsky · 1992
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams · 1992
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Convolutional learning of spatio-temporal features
Graham W Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler · 2010
Earlier work this paper cites.
3d convolutional neural networks for human action recognition
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu · 2012
Earlier work this paper cites.
Large-scale video classification with convolutional neural networks
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei · 2014
Earlier work this paper cites.
Two-stream convolutional networks for action recognition in videos
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Long-term recurrent convolutional networks for visual recognition and description
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell · 2015
Earlier work this paper cites.
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2015
Earlier work this paper cites.
Learning spatiotemporal features with 3d convolutional networks
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri · 2015
Earlier work this paper cites.
Towards good practices for very deep two-stream convnets
Original
Limin Wang, Yuanjun Xiong, Zhe Wang, and Yu Qiao · 2015
Earlier work this paper cites.
Beyond short snippets: Deep networks for video classification
Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Stochastic multiple choice learning for training diverse deep ensembles
Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David Crandall, and Dhruv Batra · 2016
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts
Original
Ilya Loshchilov and Frank Hutter · 2016
Earlier work this paper cites.
Wavenet: A generative model for raw audio
Original
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu · 2016
Earlier work this paper cites.
Hollywood in homes: Crowdsourcing data collection for activity understanding
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta · 2016
Earlier work this paper cites.
Swapout: Learning an ensemble of deep architectures
Saurabh Singh, Derek Hoiem, and David Forsyth · 2016
Earlier work this paper cites.
Temporal segment networks: Towards good practices for deep action recognition
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool · 2016
Earlier work this paper cites.
Neural architecture search with reinforcement learning
Original
Barret Zoph and Quoc V Le · 2016
Earlier work this paper cites.
Revisiting the effectiveness of off-the-shelf temporal modeling approaches for large-scale video classification
Original
Yunlong Bian, Chuang Gan, Xiao Liu, Fu Li, Xiang Long, Yandong Li, Heng Qi, Jie Zhou, Shilei Wen, and Yuanqing Lin · 2017
Earlier work this paper cites.
Quo vadis, action recognition? a new model and the kinetics dataset
Joao Carreira and Andrew Zisserman · 2017
Earlier work this paper cites.
Spatiotemporal multiplier networks for video action recognition
Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes · 2017
Earlier work this paper cites.
The” something something” video database for learning and evaluating visual common sense
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al · 2017
Earlier work this paper cites.
The kinetics human action video dataset
Original
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al · 2017
Earlier work this paper cites.
Learning spatio-temporal representation with pseudo-3d residual networks
Zhaofan Qiu, Ting Yao, and Tao Mei · 2017
Earlier work this paper cites.
Searching for activation functions
Original
Prajit Ramachandran, Barret Zoph, and Quoc V Le · 2017
Earlier work this paper cites.
Lattice long short-term memory for human action recognition
Lin Sun, Kui Jia, Kevin Chen, Dit-Yan Yeung, Bertram E Shi, and Silvio Savarese · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.

Similar

Rethinking spatiotemporal feature learning for video understanding
Original
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy · 2017
Cited alongside, same era.
Proxylessnas: Direct neural architecture search on target task and hardware
Original
Han Cai, Ligeng Zhu, and Song Han · 2018
Cited alongside, same era.
A short note about kinetics-600
Original
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman · 2018
Cited alongside, same era.
Massively parallel video networks
Joao Carreira, Viorica Patraucean, Laurent Mazare, Andrew Zisserman, and Simon Osindero · 2018
Cited alongside, same era.
Temporal modeling using dilated convolution and gating for voice-activity-detection
Shuo-Yiin Chang, Bo Li, Gabor Simko, Tara N Sainath, Anshuman Tripathi, Aäron van den Oord, and Oriol Vinyals · 2018
Cited alongside, same era.
Big-little net: An efficient multi-scale feature representation for visual and speech recognition
Original
Chun-Fu Chen, Quanfu Fan, Neil Mallinar, Tom Sercu, and Rogerio Feris · 2018
Cited alongside, same era.
Autoaugment: Learning augmentation policies from data
Original
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le · 2018
Cited alongside, same era.
Sigmoid-weighted linear units for neural network function approximation in reinforcement learning
Stefan Elfwing, Eiji Uchibe, and Kenji Doya · 2018
Cited alongside, same era.
Born again neural networks
Original
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar · 2018
Cited alongside, same era.
Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh · 2018
Cited alongside, same era.
Squeeze-and-excitation networks
Jie Hu, Li Shen, and Gang Sun · 2018
Cited alongside, same era.
Neural architecture search with bayesian optimisation and optimal transport
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing · 2018
Cited alongside, same era.

Then

Stm: Spatiotemporal and motion encoding for action recognition
Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan · 2019
Later among the works it cites.
Tsm: Temporal shift module for efficient video understanding
Ji Lin, Chuang Gan, and Song Han · 2019
Later among the works it cites.
Moments in time dataset: one million videos for event understanding
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, et al · 2019
Later among the works it cites.
Learning spatio-temporal representation with local and global diffusion
Zhaofan Qiu, Ting Yao, Chong-Wah Ngo, Xinmei Tian, and Tao Mei · 2019
Later among the works it cites.
Assemblenet: Searching for multi-stream neural connectivity in video architectures
Original
Michael S Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova · 2019
Later among the works it cites.
Mnasnet: Platform-aware neural architecture search for mobile
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le · 2019
Later among the works it cites.
Efficientnet: Rethinking model scaling for convolutional neural networks
Original
Mingxing Tan and Quoc V Le · 2019
Later among the works it cites.
Video classification with channel-separated convolutional networks
Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli · 2019
Later among the works it cites.
Rezero is all you need: Fast convergence at large depth
Original
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W Cottrell, and Julian McAuley · 2020
Later among the works it cites.
Can weight sharing outperform random architecture search? an investigation with tunas
Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoc V Le · 2020
Later among the works it cites.
Stock movement prediction that integrates heterogeneous data sources using dilated causal convolution networks with attention
Divyanshu Daiya, Min-Sheng Wu, and Che Lin · 2020
Later among the works it cites.
Rescaling egocentric vision
Original
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al · 2020
Later among the works it cites.
Rubiksnet: Learnable 3d-shift for efficient video action recognition
Linxi Fan, Shyamal Buch, Guanzhi Wang, Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei-Fei · 2020
Later among the works it cites.
X3d: Expanding architectures for efficient video recognition
Christoph Feichtenhofer · 2020
Later among the works it cites.
When ensembling smaller models is more efficient than single large models
Original
Dan Kondratyuk, Mingxing Tan, Matthew Brown, and Boqing Gong · 2020
Later among the works it cites.
Motionsqueeze: Neural motion feature learning for video understanding
Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho · 2020
Later among the works it cites.
Diverse temporal aggregation and depthwise spatiotemporal factorization for efficient video classification
Original
Youngwan Lee, Hyung-Il Kim, Kimin Yun, and Jinyoung Moon · 2020
Later among the works it cites.
Smallbignet: Integrating core and contextual views for video classification
Xianhang Li, Yali Wang, Zhipeng Zhou, and Yu Qiao · 2020
Later among the works it cites.
Perf-net: Pose empowered rgb-flow net
Original
Yinxiao Li, Zhichao Lu, Xuehan Xiong, and Jonathan Huang · 2020
Later among the works it cites.
Tam: Temporal adaptive module for video recognition
Original
Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu · 2020
Later among the works it cites.
On power laws in deep ensembles
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry Vetrov · 2020
Later among the works it cites.
Tiny video networks: Architecture search for efficient video models
AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo · 2020
Later among the works it cites.
Assemblenet++: Assembling modality representations via attention connections-supplementary material
Michael S Ryoo, AJ Piergiovanni, Juhana Kangaspunta, and Anelia Angelova · 2020
Later among the works it cites.
Learn to cycle: Time-consistent feature discovery for action recognition
Original
Alexandros Stergiou and Ronald Poppe · 2020
Later among the works it cites.
Dynamic inference: A new approach toward efficient video action recognition
Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, Yi Yang, and Shilei Wen · 2020
Later among the works it cites.
Self-training with noisy student improves imagenet classification
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le · 2020
Later among the works it cites.
Faster recurrent networks for efficient video classification
Linchao Zhu, Du Tran, Laura Sevilla-Lara, Yi Yang, Matt Feiszli, and Heng Wang · 2020
Later among the works it cites.
Vivit: A video vision transformer
Original
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid · 2021
Closest in time.
Is space-time attention all you need for video understanding?
Original
Gedas Bertasius, Heng Wang, and Lorenzo Torresani · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…