2021

DRO: Deep Recurrent Optimizer for Video to Depth

Gu, Xiaodong, Yuan, Weihao, Dai, Zuozhuo et al.

Understand

There are increasing interests of studying the video-to-depth (V2D) problem with machine learning techniques.

While earlier methods directly learn a mapping from images to depth maps and camera poses, more recent works enforce multi-view geometry constraints through optimization embedded in the learning framework.
This paper presents a novel optimization method based on recurrent neural networks to further exploit the potential of neural networks in V2D.
Specifically, our neural optimizer alternately updates the depth and camera poses through iterations to minimize a feature-metric cost, and two gated recurrent units iteratively improve the results by tracing historical information.

Built on

Z. Yu and S. Gao, “Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement,” in
1958
Earlier work this paper cites.
Z. Yin and J. Shi, “Geonet: Unsupervised learning of dense depth, optical flow and camera pose,” in
1992
Earlier work this paper cites.
Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli
2004
Earlier work this paper cites.
H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in
2011
Earlier work this paper cites.
D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in
2014
Earlier work this paper cites.
J. Engel, T. Schöps, and D. Cremers, “Lsd-slam: Large-scale direct monocular slam,” in
2014
Earlier work this paper cites.
J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,”
Original
2014
Earlier work this paper cites.
D. Jayaraman and K. Grauman, “Learning image representations tied to ego-motion,” in
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
2016
Earlier work this paper cites.
B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox, “Demon: Depth and motion network for learning monocular stereo,” in
2017
Earlier work this paper cites.
J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,”
2017
Earlier work this paper cites.
T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” in
2017
Earlier work this paper cites.
B. Amos and J. Z. Kolter, “Optnet: Differentiable optimization as a layer in neural networks,” in
2017
Earlier work this paper cites.
J. Adler and O. Öktem, “Solving ill-posed inverse problems using iterative deep neural networks,”
2017
Earlier work this paper cites.
Y. Chen, M. W. Hoffman, S. G. Colmenarejo, M. Denil, T. P. Lillicrap, M. Botvinick, and N. Freitas, “Learning to learn without gradient descent by gradient descent,” in
2017
Earlier work this paper cites.
Y. Kuznietsov, J. Stückler, and B. Leibe, “Semi-supervised deep learning for monocular depth map prediction,” in
2017
Earlier work this paper cites.

Similar

A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in
2017
Cited alongside, same era.
R. Clark, M. Bloesch, J. Czarnowski, S. Leutenegger, and A. J. Davison, “Ls-net: Learning to solve nonlinear least squares for monocular stereo,”
Original
2018
Cited alongside, same era.
M. Bloesch, J. Czarnowski, R. Clark, S. Leutenegger, and A. J. Davison, “Codeslam—learning a compact, optimisable representation for dense visual slam,” in
2018
Cited alongside, same era.
H. Zhou, B. Ummenhofer, and T. Brox, “Deeptam: Deep tracking and mapping,” in
2018
Cited alongside, same era.
Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, “Mvsnet: Depth inference for unstructured multi-view stereo,” in
2018
Cited alongside, same era.
R. Mahjourian, M. Wicke, and A. Angelova, “Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints,” in
2018
Cited alongside, same era.
H. Zhan, R. Garg, C. S. Weerasekera, K. Li, H. Agarwal, and I. Reid, “Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction,” in
2018
Cited alongside, same era.
Y. Zou, Z. Luo, and J.-B. Huang, “Df-net: Unsupervised joint learning of depth and flow using cross-task consistency,” in
2018
Cited alongside, same era.
L. Fan, W. Huang, C. Gan, S. Ermon, B. Gong, and J. Huang, “End-to-end learning of motion representation for video understanding,” in
2018
Cited alongside, same era.
C. Tang and P. Tan, “Ba-net: Dense bundle adjustment networks,” in
2019
Cited alongside, same era.
J. H. Lee, M.-K. Han, D. W. Ko, and I. H. Suh, “From big to small: Multi-scale local planar guidance for monocular depth estimation,”
Original
2019
Cited alongside, same era.
M. Bloesch, T. Laidlow, R. Clark, S. Leutenegger, and A. J. Davison, “Learning meshes for dense visual slam,” in
2019
Cited alongside, same era.

Then

J.-W. Bian, Z. Li, N. Wang, H. Zhan, C. Shen, M.-M. Cheng, and I. Reid, “Unsupervised scale-consistent depth and ego-motion learning from monocular video,” in
2019
Later among the works it cites.
A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and Z. Kolter, “Differentiable convex optimization layers,” in
2019
Later among the works it cites.
Y. Chen, C. Schmid, and C. Sminchisescu, “Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera,” in
2019
Later among the works it cites.
C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self-supervised monocular depth estimation,” in
2019
Later among the works it cites.
V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon, “3d packing for self-supervised monocular depth estimation,” in
2020
Later among the works it cites.
Z. Teed and J. Deng, “Deepv2d: Video to depth with differentiable structure from motion,” in
2020
Later among the works it cites.
X. Wei, Y. Zhang, Z. Li, Y. Fu, and X. Xue, “Deepsfm: Structure from motion via deep bundle adjustment,” in
2020
Later among the works it cites.
Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in
2020
Later among the works it cites.
X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan, “Cascade cost volume for high-resolution multi-view stereo and stereo matching,” in
2020
Later among the works it cites.
M. Klingner, J.-A. Termöhlen, J. Mikolajczyk, and T. Fingscheidt, “Self-supervised monocular depth estimation: Solving the dynamic object problem by semantic guidance,” in
2020
Later among the works it cites.
W. Zhao, S. Liu, Y. Shu, and Y.-J. Liu, “Towards better generalization: Joint depth-pose learning without posenet,” in
2020
Later among the works it cites.
J. Wang, Y. Zhong, Y. Dai, S. Birchfield, K. Zhang, N. Smolyanskiy, and H. Li, “Deep two-view structure-from-motion revisited,” in
2021
Closest in time.
W. Yuan, Y. Zhang, B. Wu, S. Zhu, P. Tan, M. Y. Wang, and Q. Chen, “Stereo matching by self-supervision of multiscopic vision,” in
2021
Closest in time.
J. Watson, O. Mac Aodha, V. Prisacariu, G. Brostow, and M. Firman, “Self-supervised multi-frame monocular depth,” in
2021
Closest in time.
W. Yuan, X. Gu, Z. Dai, S. Zhu, and P. Tan, “Newcrfs: Neural window fully-connected crfs for monocular depth estimation,” in
2022
Closest in time.
C. Wang, J. M. Buenaposada, R. Zhu, and S. Lucey, “Learning depth from monocular videos using direct methods,” in
2030
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…