2016

Optical Flow Estimation using a Spatial Pyramid Network

Ranjan, Anurag, Black, Michael J.

Understand

We learn to compute optical flow by combining a classical spatial-pyramid formulation with deep learning.

This estimates large motions in a coarse-to-fine approach by warping one image of a pair at each pyramid level by the current flow estimate and computing an update to the flow.
Instead of the standard minimization of an objective function at each pyramid level, we train one deep network per level to compute the flow update.
Unlike the recent FlowNet approach, the networks do not need to deal with large motions; these are dealt with by the pyramid.

Built on

Determining optical flow
B. K. Horn and B. G. Schunck · 1981
Earlier work this paper cites.
The Laplacian pyramid as a compact image code
P. J. Burt and E. H. Adelson · 1983
Earlier work this paper cites.
Pyramid methods in image processing
E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden · 1984
Earlier work this paper cites.
Spatiotemporal energy models for the perception of motion
E. H. Adelson and J. R. Bergen · 1985
Earlier work this paper cites.
Hierarchical motion detection
F. Glazer · 1987
Earlier work this paper cites.
Model for the extraction of image flow
D. J. Heeger · 1987
Earlier work this paper cites.
A framework for the robust estimation of optical flow
M. J. Black and P. Anandan · 1993
Earlier work this paper cites.
Performance of optical flow techniques
J. Barron, D. J. Fleet, and S. S. Beauchemin · 1994
Earlier work this paper cites.
A model of neuronal responses in visual area MT
E. P. Simoncelli and D. J. Heeger · 1998
Earlier work this paper cites.
Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex
L. van Hateren and J. Ruderman · 1998
Earlier work this paper cites.
Learning low-level vision
W. T. Freeman, E. C. Pasztor, and O. T. Carmichael · 2000
Earlier work this paper cites.
Learning sparse, overcomplete representations of time-varying natural images
B. A. Olshausen · 2003
Earlier work this paper cites.
High accuracy optical flow estimation based on a theory for warping
T. Brox, A. Bruhn, N. Papenberg, and J. Weickert · 2004
Earlier work this paper cites.
Learning transformational invariants from natural movies
C. Cadieu and B. A. Olshausen · 2008
Earlier work this paper cites.
Learning optical flow
S. D., S. Roth, J. Lewis, and M. Black · 2008
Earlier work this paper cites.

Similar

Large displacement optical flow
T. Brox, C. Bregler, and J. Malik · 2009
Cited alongside, same era.
Fields of experts
S. Roth and M. J. Black · 2009
Cited alongside, same era.
Anisotropic Huber-L1 optical flow
M. Werlberger, W. Trobin, T. Pock, A. Wedel, D. Cremers, and H. Bischof · 2009
Cited alongside, same era.
Learning to represent spatial transformations with factored higher-order boltzmann machines
R. Memisevic and G. E. Hinton · 2010
Cited alongside, same era.
Dense point trajectories by gpu-accelerated large displacement optical flow
N. Sundaram, T. Brox, and K. Keutzer · 2010
Cited alongside, same era.
Convolutional learning of spatio-temporal features
G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler · 2010
Cited alongside, same era.
A database and evaluation methodology for optical flow
S. Baker, D. Scharstein, J. Lewis, S. Roth, M. J. Black, and R. Szeliski · 2011
Cited alongside, same era.
A naturalistic open source movie for optical flow evaluation
D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black · 2012
Cited alongside, same era.
Are we ready for autonomous driving? the KITTI vision benchmark suite
A. Geiger, P. Lenz, and R. Urtasun · 2012
Cited alongside, same era.
Deepflow: Large displacement optical flow with deep matching
P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid · 2013
Cited alongside, same era.
Fast edge-preserving PatchMatch for large displacement optical flow
L. Bao, Q. Yang, and H. Jin · 2014
Cited alongside, same era.
Semantic image segmentation with deep convolutional nets and fully connected crfs
Original
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille · 2014
Cited alongside, same era.

Then

Deep generative image models using a laplacian pyramid of adversarial networks
E. L. Denton, S. Chintala, R. Fergus, et al · 2015
Later among the works it cites.
Flownet: Learning optical flow with convolutional networks
A. Dosovitskiy, P. Fischery, E. Ilg, C. Hazirbas, V. Golkov, P. van der Smagt, D. Cremers, T. Brox, et al · 2015
Later among the works it cites.
Scalable robust principal component analysis using Grassmann averages
S. Hauberg, A. Feragen, R. Enficiaud, and M. Black · 2015
Later among the works it cites.
Deep residual learning for image recognition
Original
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Later among the works it cites.
Fully convolutional networks for semantic segmentation
J. Long, E. Shelhamer, and T. Darrell · 2015
Later among the works it cites.
EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow
J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid · 2015
Later among the works it cites.
ImageNet Large Scale Visual Recognition Challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Later among the works it cites.
Going deeper with convolutions
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich · 2015
Later among the works it cites.
Efficient sparse-to-dense optical flow estimation using a learned basis and layers
J. Wulff and M. J. Black · 2015
Later among the works it cites.
Unsupervised convolutional neural networks for motion estimation
Original
A. Ahmadi and I. Patras · 2016
Closest in time.
Deep discrete flow
F. Güney and A. Geiger · 2016
Closest in time.
Fast optical flow using dense inverse search
T. Kroeger, R. Timofte, D. Dai, and L. V. Gool · 2016
Closest in time.
A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation
Original
N.Mayer, E.Ilg, P.Häusser, P.Fischer, D.Cremers, A.Dosovitskiy, and T.Brox · 2016
Closest in time.
Deep End2End Voxel2Voxel prediction
D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri · 2016
Closest in time.
Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness
Original
J. J. Yu, A. W. Harley, and K. G. Derpanis · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…