2016

Optical Flow with Semantic Segmentation and Localized Layers

Sevilla-Lara, Laura, Sun, Deqing, Jampani, Varun et al.

Understand

Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow.

In reality, optical flow varies across an image depending on object class.
Simply put, different objects move differently.
Here we exploit recent advances in static semantic scene segmentation to segment the image into objects of different types.

Built on

Scene segmentation from visual motion using global optimization
D. W. Murray and B. F. Buxton · 1987
Earlier work this paper cites.
Dense estimation and object-based segmentation of the optical flow with robust techniques
E. Mémin and P. Pérez · 1988
Earlier work this paper cites.
The plenoptic function and the elements of early vision
E. Adelson and J. Bergen · 1991
Earlier work this paper cites.
Robust estimation of a multi-layered motion representation
T. Darrell and A. Pentland · 1991
Earlier work this paper cites.
Mixture models for optical flow computation
A. Jepson and M. J. Black · 1993
Earlier work this paper cites.
Accurate computation of optical flow by using layered motion representations
S. Hsu, P. Anandan, and S. Peleg · 1994
Earlier work this paper cites.
Representing moving images with layers
J. Y. A. Wang and E. H. Adelson · 1994
Earlier work this paper cites.
Layered representation of motion video using robust maximum-likelihood estimation of mixture models and MDL encoding
S. Ayer and H. S. Sawhney · 1995
Earlier work this paper cites.
Estimating optical flow in segmented images using variable-order parametric models with local deformations
M. J. Black and A. Jepson · 1996
Earlier work this paper cites.
Skin and bones: Multi-layer, locally affine, optical flow and regularization with transparency
S. Ju, M. J. Black, and A. D. Jepson · 1996
Earlier work this paper cites.
Smoothness in layers: Motion segmentation using nonparametric mixture estimation
Y. Weiss · 1997
Earlier work this paper cites.
A unified approach to moving object detection in 2d and 3d scenes
M. Irani and P. Anandan · 1998
Earlier work this paper cites.
Exploiting discontinuities in optical flow
W. B. Thompson · 1998
Earlier work this paper cites.
Probabilistic detection and tracking of motion boundaries
M. J. Black and D. J. Fleet · 2000
Earlier work this paper cites.
On seeing stuff: the perception of materials by humans and machines
E. H. Adelson · 2001
Earlier work this paper cites.
Learning flexible sprites in video layers
N. Jojic and B. Frey · 2001
Earlier work this paper cites.
A layered motion representation with occlusion and compact spatial support
A. Jepson, D. Fleet, and M. Black · 2002
Earlier work this paper cites.
A background layer model for object tracking through occlusion
Y. Zhou and H. Tao · 2003
Earlier work this paper cites.
Learning spatial context: Using stuff to find things
G. Heitz and D. Koller · 2008
Earlier work this paper cites.
Learning layered motion segmentations of video
M. Kumar, P. Torr, and A. Zisserman · 2008
Earlier work this paper cites.

Similar

Fusionflow: Discrete-continuous optimization for optical flow estimation
V. Lempitsky, S. Roth, and C. Rother · 2008
Cited alongside, same era.
A wavelet tour of signal processing: the sparse way
S. Mallat · 2008
Cited alongside, same era.
Occlusion boundaries from motion: Low-level detection and mid-level reasoning
A. N. Stein and M. Hebert · 2009
Cited alongside, same era.
Segmentation, ordering and multi-object tracking using graphical models
C. Wang, M. de La Gorce, and N. Paragios · 2009
Cited alongside, same era.
Efficient hierarchical graph-based video segmentation
M. Grundmann, V. Kwatra, M. Han, and I. Essa · 2010
Cited alongside, same era.
A database and evaluation methodology for optical flow
S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski · 2011
Cited alongside, same era.
Efficient inference in fully connected CRFs with gaussian edge potentials
P. Krähenbühl and V. Koltun · 2011
Cited alongside, same era.
Joint optimization for object class segmentation and dense stereo reconstruction
L. Ladický, P. Sturgess, C. Russell, S. Sengupta, Y. Bastanlar, W. Clocksin, and P. H. S. Torr · 2011
Cited alongside, same era.
Object flow: Learning object displacement
C. Lalos, H. Grabner, L. Van Gool, and T. Varvarigou · 2011
Cited alongside, same era.
Occlusion boundary detection and figure/ground assignment from optical flow
P. Sundberg, T. Brox, M. Maire, P. Arbelaez, and J. Malik · 2011
Cited alongside, same era.
A naturalistic open source movie for optical flow evaluation
D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black · 2012
Cited alongside, same era.
Efficient nonlocal regularization for optical flow
P. Krähenbühl and V. Koltun · 2012
Cited alongside, same era.

Then

Semantic video segmentation from occlusion relations within a convex optimization framework
B. Taylor, A. Ayvaci, A. Ravichandran, and S. Soatto · 2013
Later among the works it cites.
Robust monocular epipolar flow estimation
K. Yamaguchi, D. McAllester, and R. Urtasun · 2013
Later among the works it cites.
Semantic image segmentation with deep convolutional nets and fully connected crfs
Original
L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille · 2014
Later among the works it cites.
Caffe: Convolutional architecture for fast feature embedding
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell · 2014
Later among the works it cites.
The role of context for object detection and semantic segmentation in the wild
R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille · 2014
Later among the works it cites.
Segmentation of moving objects by long term video analysis
P. Ochs, J. Malik, and T. Brox · 2014
Later among the works it cites.
Stixmantics: A medium-level model for real-time semantic scene understanding
T. Scharwächter, M. Enzweiler, U. Franke, and S. Roth · 2014
Later among the works it cites.
Very deep convolutional networks for large-scale image recognition
Original
K. Simonyan and A. Zisserman · 2014
Later among the works it cites.
Local layering for joint motion estimation and occlusion detection
D. Sun, C. Liu, and H. Pfister · 2014
Later among the works it cites.
Displets: Resolving stereo ambiguities using object knowledge
F. Güney and A. Geiger · 2015
Later among the works it cites.
Scene collaging: Analysis and synthesis of natural images with semantic layers
P. Isola and C. Liu · 2015
Later among the works it cites.
Multiclass semantic video segmentation with object-level active inference
B. Liu and X. He · 2015
Later among the works it cites.
Fully convolutional networks for semantic segmentation
J. Long, E. Shelhamer, and T. Darrell · 2015
Later among the works it cites.
Object scene flow for autonomous vehicles
M. Menze and A. Geiger · 2015
Later among the works it cites.
Discrete optimization for optical flow
M. Menze, C. Heipke, and A. Geiger · 2015
Later among the works it cites.
EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow
J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid · 2015
Later among the works it cites.
Efficient sparse-to-dense optical flow estimation using a learned basis and layers
J. Wulff and M. J. Black · 2015
Later among the works it cites.
Dense, accurate optical flow estimation with piecewise parametric model
J. Yang and H. Li · 2015
Later among the works it cites.
Full flow: Optical flow estimation by global optimization over regular grids
Q. Chen and V. Koltun · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…