2019

Hallucinating IDT Descriptors and I3D Optical Flow Features for Action Recognition with CNNs

Wang, Lei, Koniusz, Piotr, Huynh, Du Q.

Understand

In this paper, we revive the use of old-fashioned handcrafted video representations for action recognition and put new life into these techniques via a CNN-based hallucination step.

Despite of the use of RGB and optical flow frames, the I3D model (amongst others) thrives on combining its output with the Improved Dense Trajectory (IDT) and extracted with its low-level video descriptors encoded via Bag-of-Words (BoW) and Fisher Vectors (FV).
Such a fusion of CNNs and handcrafted representations is time-consuming due to pre-processing, descriptor extraction, encoding and tuning parameters.
Thus, we propose an end-to-end trainable network with streams which learn the IDT-based BoW/FV representations at the training stage and are simple to integrate with the I3D model.

Built on

Book review: Thinking in perspective: Critical essays in the study of thought processes
Ian M. L. Hunter · 1980
Earlier work this paper cites.
Determining optical flow
Berthold K. P. Horn and Brian G. Schunck · 1981
Earlier work this paper cites.
Orientation histograms for hand gesture recognition
William T. Freeman and Michal Roth · 1994
Earlier work this paper cites.
Is learning the n-th thing any easier than learning the first?
Sebastian Thrun · 1996
Earlier work this paper cites.
Multitask learning
Rich Caruana · 1997
Earlier work this paper cites.
Reliable estimation of dense optical flow fields with large displacements
Luis Alvarez, Joachim Weickert, and Javier Sánchez · 2000
Earlier work this paper cites.
Video Google: A text retrieval approach to object matching in videos
Josef Sivic and Andrew Zisserman · 2003
Earlier work this paper cites.
Visual categorization with bags of keypoints
Gabriella Csurka, Christopher R. Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray · 2004
Earlier work this paper cites.
Behavior recognition via sparse spatio-temporal features
Piotr Dollár, Vincent Rabaud, Garrison Cottrell, and Serge Belongie · 2005
Earlier work this paper cites.
On space-time interest points
Ivan Laptev · 2005
Earlier work this paper cites.
Human Detection Using Oriented Histogram of Flow and Appearance
Navneet Dalal, Bill Triggs, and Cordelia Schmid · 2006
Earlier work this paper cites.
Highly accurate optic flow computation with theoretically justified warping
Nils Papenberg, Andrés Bruhn, Thomas Brox, Stephan Didas, and Joachim Weickert · 2006
Earlier work this paper cites.
Fisher kernels on visual vocabularies for image categorization
Florent Perronnin and Christopher Dance · 2007
Earlier work this paper cites.
A 3-Dimentional SIFT Descriptor and its Application to Action Recognition
Paul Scovanner, Saad Ali, and Mubarak Shah · 2007
Earlier work this paper cites.
Finding frequent items in data streams
Graham Cormode and Marios Hadjieleftheriou · 2008
Earlier work this paper cites.
A Spatio-Temporal Descriptor Based on 3D-Gradients
Alexander Kläser, Marcin Marszalek, and Cordelia Schmid · 2008
Earlier work this paper cites.
An efficient dense and scale-invariant spatio-temporal interest point detector
Geert Willems, Tinne Tuytelaars, and Luc Van Gool · 2008
Earlier work this paper cites.
On the Burstiness of Visual Elements
Hervé Jégou, Matthijs Douze, and Cordelia Schmid · 2009
Earlier work this paper cites.
Feature hashing for large scale multitask learning
Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg · 2009
Earlier work this paper cites.
Local trinary patterns for human action recognition
Lahav Yeffet and Lior Wolf · 2009
Earlier work this paper cites.
3D convolutional neural networks for human action recognition
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu · 2010
Earlier work this paper cites.
Improving the Fisher Kernel for Large-Scale Image Classification
Florent Perronnin, Jorge Sánchez, and Thomas Mensink · 2010
Earlier work this paper cites.
Visual word ambiguity
Jan C. van Gemert, Cor J. Veenman, Arnold W. M. Smeulders, and Jan-Mark Geusebroek · 2010
Earlier work this paper cites.
Large displacement optical flow: Descriptor matching in variational motion estimation
Thomas Brox and Jitendra Malik · 2011
Earlier work this paper cites.
Evaluation of interest point detectors and feature descriptors for visual tracking
Steffen Gauglitz, Tobias Höllerer, and Matthew Turk · 2011
Earlier work this paper cites.

Similar

Soft Assignment of Visual Words as Linear Coordinate Coding and Optimisation of its Reconstruction Error
Piotr Koniusz and Krystian Mikolajczyk · 2011
Cited alongside, same era.
HMDB: A large video database for human motion recognition
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre · 2011
Cited alongside, same era.
In Defence of Soft-assignment Coding
Liu Lingqiao, Lei Wang, and Xinwang Liu · 2011
Cited alongside, same era.
Action Recognition by Dense Trajectories
Heng Wang, Alexander Kläser, Cordelia Schmid, and Liu Cheng-Lin · 2011
Cited alongside, same era.
Selective spatio-temporal interest points
Bhaskar Chakraborty, Michael B. Holte, Thomas B. Moeslund, and Jordi Gonzàlez · 2012
Cited alongside, same era.
Comparison of Mid-Level Feature Coding Approaches And Pooling Strategies in Visual Concept Detection
Piotr Koniusz, Fei Yan, and Krystian Mikolajczyk · 2012
Cited alongside, same era.
ImageNet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton · 2012
Cited alongside, same era.
A database for fine grained activity detection of cooking activities
Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele · 2012
Cited alongside, same era.
A general dense image matching framework combining direct and feature-based costs
Jim Braux-Zin, Romain Dupont, and Adrien Bartoli · 2013
Cited alongside, same era.
Higher-order Occurrence Pooling on Mid- and Low-level Features: Visual Concept Detection
Piotr Koniusz, Fei Yan, Philippe-Henri Gosselin, and Krystian Mikolajczyk · 2013
Cited alongside, same era.
Fast and scalable polynomial kernels via explicit feature maps
Ninh Pham and Rasmus Pagh · 2013
Cited alongside, same era.
Dense Trajectories and Motion Boundary Descriptors for Action Recognition
Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng-Lin Liu · 2013
Cited alongside, same era.

Then

Learning end-to-end video classification with rank-pooling
Basura Fernando and Stephen Gould · 2016
Later among the works it cites.
Tensor representations via kernel linearization for action recognition from 3D skeletons
Piotr Koniusz, Anoop Cherian, and Fatih Porikli · 2016
Later among the works it cites.
Hollywood in homes: Crowdsourcing data collection for activity understanding
Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta · 2016
Later among the works it cites.
Generalized rank pooling for action recognition
Anoop Cherian, Basura Fernando, Mehrtash Harandi, and Stephen Gould · 2017
Later among the works it cites.
Higher-order pooling of CNN features via kernel linearization for action recognition
Anoop Cherian, Piotr Koniusz, and Stephen Gould · 2017
Later among the works it cites.
Spatiotemporal multiplier networks for video action recognition
Christoph Feichtenhofer, Axel Pinz, and Richard P. Wildes · 2017
Later among the works it cites.
Temporal residual networks for dynamic scene recognition
Christoph Feichtenhofer, Axel Pinz, and Richard P. Wildes · 2017
Later among the works it cites.
Domain adaptation by mixture of alignments of second- or higher-order scatter tensors
Piotr Koniusz, Yusuf Tas, and Fatih Porikli · 2017
Later among the works it cites.
Higher-order occurrence pooling for bags-of-words: Visual concept detection
Piotr Koniusz, Fei Yan, Philippe-Henri Gosselin, and Krystian Mikolajczyk · 2017
Later among the works it cites.
Analysis and evaluation of Kinect-based action recognition algorithms
Lei Wang · 2017
Later among the works it cites.
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
João Carreira and Andrew Zisserman · 2018
Later among the works it cites.
Non-linear temporal subspace representations for activity recognition
Anoop Cherian, Suvrit Sra, Stephen Gould, and Richard Hartley · 2018
Later among the works it cites.
PoTion: Pose motion representation for action recognition
Vasileios Choutas, Philippe Weinzaepfel, Jérôme Revaud, and Cordelia Schmid · 2018
Later among the works it cites.
Dimensionality reduction on SPD manifolds: The emergence of geometry-aware methods
Mehrtash Harandi, Mathieu Salzmann, and Richard Hartley · 2018
Later among the works it cites.
Museum exhibit identification challenge for the supervised domain adaptation and beyond
Piotr Koniusz, Yusuf Tas, Hongguang Zhang, Mehrtash Harandi, Fatih Porikli, and Rui Zhang · 2018
Later among the works it cites.
A deeper look at power normalizations
Piotr Koniusz, Hongguang Zhang, and Fatih Porikli · 2018
Later among the works it cites.
CNN-based action recognition and supervised domain adaptation on 3D body skeletons via kernel feature maps
Yusuf Tas and Piotr Koniusz · 2018
Later among the works it cites.
Long-term temporal convolutions for action recognition
Gül Varol, Ivan Laptev, and Cordelia Schmid · 2018
Later among the works it cites.
Learning discriminative video representations using adversarial perturbations
Jue Wang and Anoop Cherian · 2018
Later among the works it cites.
Bilinear attention networks for person retrieval
Pengfei Fang, Jieming Zhou, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi · 2019
Closest in time.
Min-max statistical alignment for transfer learning
Samitha Herath, Mehrtash Harandi, Basura Fernando, and Richard Nock · 2019
Closest in time.
Siamese networks: The tale of two manifolds
Soumava Kumar Roy, Mehrtash Harandi, Richard Nock, and Richard Hartley · 2019
Closest in time.
A comparative review of recent kinect-based action recognition algorithms
Lei Wang, Du Q. Huynh, and Piotr Koniusz · 2019
Closest in time.
Loss switching fusion with similarity search for video classification
Lei Wang, Du Q. Huynh, and Moussa Reda Mansour · 2019
Closest in time.
Long-term feature banks for detailed video understanding
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…