2017

Semantic Video CNNs through Representation Warping

Gadde, Raghudeep, Jampani, Varun, Gehler, Peter V.

Understand

In this work, we propose a technique to convert CNN models for semantic segmentation of static images into CNNs for video data.

We describe a warping method that can be used to augment existing architectures with very little extra computational cost.
This module is called NetWarp and we demonstrate its use for a range of network architectures.
The main design principle is to use optical flow of adjacent frames for warping internal network representations across time.

Built on

Video matting of complex scenes
Y.-Y. Chuang, A. Agarwala, B. Curless, D. H. Salesin, and R. Szeliski · 2002
Earlier work this paper cites.
Video object cut and paste
Y. Li, J. Sun, and H.-Y. Shum · 2005
Earlier work this paper cites.
Interactive video cutout
J. Wang, P. Bhat, R. A. Colburn, M. Agrawala, and M. F. Cohen · 2005
Earlier work this paper cites.
Segmentation and recognition using structure from motion point clouds
G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla · 2008
Earlier work this paper cites.
Semantic object classes in video: A high-definition ground truth database
G. J. Brostow, J. Fauqueur, and R. Cipolla · 2009
Earlier work this paper cites.
Segmentation-based urban traffic scene understanding
A. Ess, T. Mueller, H. Grabner, and L. J. Van Gool · 2009
Earlier work this paper cites.
Robust higher order potentials for enforcing label consistency
P. Kohli, P. H. Torr, et al · 2009
Earlier work this paper cites.
Livecut: Learning-based interactive video segmentation by evaluation of multiple propagated cues
B. L. Price, B. S. Morse, and S. Cohen · 2009
Earlier work this paper cites.
Combining appearance and structure from motion features for road scene understanding
P. Sturgess, K. Alahari, L. Ladicky, and P. H. Torr · 2009
Earlier work this paper cites.
Temporally consistent multi-class video-object segmentation with the video graph-shifts algorithm
A. Y. Chen and J. J. Corso · 2011
Earlier work this paper cites.
Efficient inference in fully connected crfs with gaussian edge potentials
P. Krähenbühl and V. Koltun · 2011
Earlier work this paper cites.
On-line semantic perception using uncertainty
R. de Nijs, S. Ramos, G. Roig, X. Boix, L. Van Gool, and K. Kühnlenz · 2012
Earlier work this paper cites.
Joint 2d-3d temporally consistent semantic segmentation of street scenes
G. Floros and B. Leibe · 2012
Earlier work this paper cites.
3d convolutional neural networks for human action recognition
S. Ji, W. Xu, M. Yang, and K. Yu · 2013
Earlier work this paper cites.
Efficient temporal consistency for streaming video scene analysis
O. Miksik, D. Munoz, J. A. Bagnell, and M. Hebert · 2013
Earlier work this paper cites.
Urban 3d semantic modelling using stereo vision
S. Sengupta, E. Greveson, A. Shahrokni, and P. H. Torr · 2013
Earlier work this paper cites.

Similar

Semantic image segmentation with deep convolutional nets and fully connected crfs
Original
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille · 2014
Cited alongside, same era.
Supervoxel-consistent foreground propagation in video
S. D. Jain and K. Grauman · 2014
Cited alongside, same era.
Caffe: Convolutional architecture for fast feature embedding
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell · 2014
Cited alongside, same era.
Large-scale video classification with convolutional neural networks
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
D. Kingma and J. Ba · 2014
Cited alongside, same era.
Joint semantic segmentation and 3d reconstruction from monocular video
A. Kundu, Y. Li, F. Dellaert, F. Li, and J. M. Rehg · 2014
Cited alongside, same era.
Interactive segmentation of high-resolution video content using temporally coherent superpixels and graph cut
M. Reso, B. Scheuermann, J. Jachalsky, B. Rosenhahn, and J. Ostermann · 2014
Cited alongside, same era.
Two-stream convolutional networks for action recognition in videos
K. Simonyan and A. Zisserman · 2014
Cited alongside, same era.
Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation
C. Bailer, B. Taetz, and D. Stricker · 2015
Cited alongside, same era.
Long-term recurrent convolutional networks for visual recognition and description
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell · 2015
Cited alongside, same era.
Jumpcut: non-successive mask transfer and interpolation for video cutout
Q. Fan, F. Zhong, D. Lischinski, D. Cohen-Or, and B. Chen · 2015
Cited alongside, same era.
Multi-class semantic video segmentation with exemplar-based object reasoning
B. Liu, X. He, and S. Gould · 2015
Cited alongside, same era.

Then

Laplacian pyramid reconstruction and refinement for semantic segmentation
G. Ghiasi and C. C. Fowlkes · 2016
Later among the works it cites.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Later among the works it cites.
Joint optical flow and temporally consistent semantic segmentation
J. Hur and S. Roth · 2016
Later among the works it cites.
Learning sparse high dimensional filters: Image filtering, dense crfs and bilateral neural networks
V. Jampani, M. Kiefel, and P. V. Gehler · 2016
Later among the works it cites.
Fast optical flow using dense inverse search
T. Kroeger, R. Timofte, D. Dai, and L. V. Gool · 2016
Later among the works it cites.
Feature space optimization for semantic video segmentation
A. Kundu, V. Vineet, and V. Koltun · 2016
Later among the works it cites.
Recurrent temporal deep field for semantic video labeling
P. Lei and S. Todorovic · 2016
Later among the works it cites.
Playing for data: Ground truth from computer games
S. R. Richter, V. Vineet, S. Roth, and V. Koltun · 2016
Later among the works it cites.
Clockwork convnets for video semantic segmentation
Original
E. Shelhamer, K. Rakelly, J. Hoffman, and T. Darrell · 2016
Later among the works it cites.
Temporal segment networks: towards good practices for deep action recognition
L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool · 2016
Later among the works it cites.
Wider or deeper: Revisiting the resnet model for visual recognition
Original
Z. Wu, C. Shen, and A. v. d. Hengel · 2016
Later among the works it cites.
Multi-scale context aggregation by dilated convolutions
F. Yu and V. Koltun · 2016
Later among the works it cites.
Pyramid scene parsing network
Original
H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia · 2016
Later among the works it cites.
Video propagation networks
V. Jampani, R. Gadde, and P. V. Gehler · 2017
Closest in time.
RefineNet: Multi-path refinement networks for high-resolution semantic segmentation
G. Lin, A. Milan, C. Shen, and I. Reid · 2017
Closest in time.
Understanding convolution for semantic segmentation
Original
P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou, and G. Cottrell · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…