2022

DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation

Li, Zhenyu, Chen, Zehui, Liu, Xianming et al.

Understand

This paper aims to address the problem of supervised monocular depth estimation.

We start with a meticulous pilot study to demonstrate that the long-range correlation is essential for accurate depth estimation.
Therefore, we propose to leverage the Transformer to model this global context with an effective attention mechanism.
We also adopt an additional convolution branch to preserve the local information as the Transformer lacks the spatial inductive bias in modeling such contents.

Built on

Saxena A, Chung SH, Ng AY, et al. (2005) Learning depth from single monocular images. In: Advances in Neural Information Processing Systems (NIPS), vol 18, pp 1–8
2005
Earlier work this paper cites.
Fu H, Gong M, Wang C, Batmanghelich K, Tao D (2018) Deep ordinal regression network for monocular depth estimation. In: Computer Vision and Pattern Recognition (CVPR), pp 2002–2011
2011
Earlier work this paper cites.
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NIPS) 25:1097–1105
2012
Earlier work this paper cites.
Silberman N, Hoiem D, Kohli P, Fergus R (2012) Indoor segmentation and support inference from rgbd images. In: European Conference on Computer Vision (ECCV), pp 746–760
2012
Earlier work this paper cites.
Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: The kitti dataset. The International Journal of Robotics Research 32(11):1231–1237
2013
Earlier work this paper cites.
Janoch A, Karayev S, Jia Y, Barron JT, Fritz M, Saenko K, Darrell T (2013) A category-level 3d object dataset: Putting the kinect to work. In: Consumer depth cameras for computer vision, Springer, pp 141–165
2013
Earlier work this paper cites.
Xiao J, Owens A, Torralba A (2013) Sun3d: A database of big spaces reconstructed using sfm and object labels. In: International Conference on Computer Vision (ICCV), pp 1625–1632
2013
Earlier work this paper cites.
Eigen D, Puhrsch C, Fergus R (2014) Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems (NIPS)
2014
Earlier work this paper cites.
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp 234–241
2015
Earlier work this paper cites.
Song S, Lichtenberg SP, Xiao J (2015) Sun rgb-d: A rgb-d scene understanding benchmark suite. In: Computer Vision and Pattern Recognition (CVPR), pp 567–576
2015
Earlier work this paper cites.
Garg R, Bg VK, Carneiro G, Reid I (2016) Unsupervised cnn for single view depth estimation: Geometry to the rescue. In: European Conference on Computer Vision (ECCV), pp 740–756
2016
Earlier work this paper cites.
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition (CVPR), pp 770–778
2016
Earlier work this paper cites.
Laina I, Rupprecht C, Belagiannis V, Tombari F, Navab N (2016) Deeper depth prediction with fully convolutional residual networks. In: International Conference on 3D Vision (3DV), pp 239–248
2016
Earlier work this paper cites.
Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. Transactions on Pattern Analysis and Machine Intelligence 40(4):834–848
2017
Earlier work this paper cites.
Dai J, Qi H, Xiong Y, Li Y, Zhang G, Hu H, Wei Y (2017) Deformable convolutional networks. In: International Conference on Computer Vision (ICCV), pp 764–773
2017
Earlier work this paper cites.

Similar

Godard C, Mac Aodha O, Brostow GJ (2017) Unsupervised monocular depth estimation with left-right consistency. In: Computer Vision and Pattern Recognition (CVPR), pp 270–279
2017
Cited alongside, same era.
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Computer Vision and Pattern Recognition (CVPR), pp 4700–4708
2017
Cited alongside, same era.
Uhrig J, Schneider N, Schneider L, Franke U, Brox T, Geiger A (2017) Sparsity invariant cnns. In: International Conference on 3D Vision (3DV), IEEE, pp 11–20
2017
Cited alongside, same era.
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in Neural Information Processing Systems (NIPS), pp 5998–6008
2017
Cited alongside, same era.
Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Computer Vision and Pattern Recognition (CVPR), pp 2881–2890
2017
Cited alongside, same era.
Gan Y, Xu X, Sun W, Lin L (2018) Monocular depth estimation with affinity, vertical pooling, and label enhancement. In: European Conference on Computer Vision (ECCV), pp 224–239
2018
Cited alongside, same era.
Jiao J, Cao Y, Song Y, Lau R (2018) Look deeper into depth: Monocular depth estimation with semantic booster and attention-driven loss. In: European Conference on Computer Vision (ECCV), pp 53–69
2018
Cited alongside, same era.
Chen X, Chen X, Zha ZJ (2019) Structure-aware residual pyramid network for monocular depth estimation. International Joint Conference on Artificial Intelligence (IJCAI) pp 694–700
2019
Cited alongside, same era.
Hu J, Ozay M, Zhang Y, Okatani T (2019) Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries. In: Winter Conference on Applications of Computer Vision (WACV), pp 1043–1051
2019
Cited alongside, same era.
Lee JH, Han MK, Ko DW, Suh IH (2019) From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv:190710326
2019
Cited alongside, same era.
Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning (ICML), pp 6105–6114
2019
Cited alongside, same era.
Yin W, Liu Y, Shen C, Yan Y (2019) Enforcing geometric constraints of virtual normal for depth prediction. In: International Conference on Computer Vision (ICCV), pp 5684–5693
2019
Cited alongside, same era.

Then

Huynh L, Nguyen-Ha P, Matas J, Rahtu E, Heikkilä J (2020) Guiding monocular depth estimation using depth-attention volume. In: European Conference on Computer Vision (ECCV), pp 581–597
2020
Later among the works it cites.
Johnston A, Carneiro G (2020) Self-supervised monocular trained depth estimation using self-attention and discrete disparity volume. In: Computer Vision and Pattern Recognition (CVPR), pp 4756–4765
2020
Later among the works it cites.
Kolesnikov A, Beyer L, Zhai X, Puigcerver J, Yung J, Gelly S, Houlsby N (2020) Big transfer (bit): General visual representation learning. In: European Conference on Computer Vision (ECCV), pp 491–507
2020
Later among the works it cites.
Xu D, Alameda-Pineda X, Ouyang W, Ricci E, Wang X, Sebe N (2020) Probabilistic graph attention network with conditional kernels for pixel-wise prediction. Transactions on Pattern Analysis and Machine Intelligence
2020
Later among the works it cites.
Bhat SF, Alhashim I, Wonka P (2021) Adabins: Depth estimation using adaptive bins. In: Computer Vision and Pattern Recognition (CVPR), pp 4009–4018
2021
Later among the works it cites.
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al. (2021) An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR)
2021
Later among the works it cites.
Ji P, Li R, Bhanu B, Xu Y (2021) Monoindoor: Towards good practice of self-supervised monocular depth estimation for indoor environments. In: International Conference on Computer Vision (ICCV), pp 12787–12796
2021
Later among the works it cites.
Lee S, Lee J, Kim B, Yi E, Kim J (2021) Patch-wise attention network for monocular depth estimation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 35, pp 1873–1881
2021
Later among the works it cites.
Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. International Conference on Computer Vision (ICCV)
2021
Later among the works it cites.
Qiao S, Zhu Y, Adam H, Yuille A, Chen LC (2021) Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation. In: Computer Vision and Pattern Recognition (CVPR), pp 3997–4008
2021
Later among the works it cites.
Ranftl R, Bochkovskiy A, Koltun V (2021) Vision transformers for dense prediction. In: International Conference on Computer Vision (ICCV), pp 12179–12188
2021
Later among the works it cites.
Yang G, Tang H, Ding M, Sebe N, Ricci E (2021) Transformers solve the limited receptive field for monocular depth prediction. In: International Conference on Computer Vision (ICCV)
2021
Later among the works it cites.
Zheng S, Lu J, Zhao H, Zhu X, Luo Z, Wang Y, Fu Y, Feng J, Xiang T, Torr PH, et al. (2021) Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Computer Vision and Pattern Recognition (CVPR), pp 6881–6890
2021
Later among the works it cites.
Zhu X, Su W, Lu L, Li B, Wang X, Dai J (2021) Deformable detr: Deformable transformers for end-to-end object detection. In: International Conference on Learning Representations (ICLR)
2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…