2021

DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras

Teed, Zachary, Deng, Jia

Understand

We introduce DROID-SLAM, a new deep learning based SLAM system.

DROID-SLAM consists of recurrent iterative updates of camera pose and pixelwise depth through a Dense Bundle Adjustment layer.
DROID-SLAM is accurate, achieving large improvements over prior work, and robust, suffering from substantially fewer catastrophic failures.
Despite training on monocular video, it can leverage stereo or RGB-D video to achieve improved performance at test time.

Built on

Monoslam: Real-time single camera slam
A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse · 2007
Earlier work this paper cites.
A multi-state constraint kalman filter for vision-aided inertial navigation
A. I. Mourikis and S. I. Roumeliotis · 2007
Earlier work this paper cites.
Dtam: Dense tracking and mapping in real-time
R. A. Newcombe, S. J. Lovegrove, and A. J. Davison · 2011
Earlier work this paper cites.
A naturalistic open source movie for optical flow evaluation
D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black · 2012
Earlier work this paper cites.
Indoor segmentation and support inference from rgbd images
P. K. Nathan Silberman, Derek Hoiem and R. Fergus · 2012
Earlier work this paper cites.
A benchmark for the evaluation of rgb-d slam systems
J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers · 2012
Earlier work this paper cites.
Dense visual slam for rgb-d cameras
C. Kerl, J. Sturm, and D. Cremers · 2013
Earlier work this paper cites.
Lsd-slam: Large-scale direct monocular slam
J. Engel, T. Schöps, and D. Cremers · 2014
Earlier work this paper cites.
Large-scale direct slam for omnidirectional cameras
D. Caruso, J. Engel, and D. Cremers · 2015
Earlier work this paper cites.
Multi-view stereo: A tutorial
Y. Furukawa and C. Hernández · 2015
Earlier work this paper cites.
Orb-slam: a versatile and accurate monocular slam system
R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos · 2015
Earlier work this paper cites.
Real-time large-scale dense rgb-d slam with volumetric fusion
T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald · 2015
Earlier work this paper cites.
Elasticfusion: Dense slam without a pose graph
T. Whelan, S. Leutenegger, R. Salas-Moreno, B. Glocker, and A. Davison · 2015
Earlier work this paper cites.
The euroc micro aerial vehicle datasets
M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart · 2016
Earlier work this paper cites.
Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age
C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard · 2016
Earlier work this paper cites.
Universal correspondence network
Original
C. B. Choy, J. Gwak, S. Savarese, and M. Chandraker · 2016
Earlier work this paper cites.
Svo: Semidirect visual odometry for monocular and multicamera systems
C. Forster, Z. Zhang, M. Gassner, M. Werlberger, and D. Scaramuzza · 2016
Earlier work this paper cites.
Structure-from-motion revisited
J. L. Schonberger and J.-M. Frahm · 2016
Earlier work this paper cites.
Scannet: Richly-annotated 3d reconstructions of indoor scenes
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner · 2017
Earlier work this paper cites.
Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration
A. Dai, M. Nießner, M. Zollhöfer, S. Izadi, and C. Theobalt · 2017
Earlier work this paper cites.
Direct sparse odometry
J. Engel, V. Koltun, and D. Cremers · 2017
Earlier work this paper cites.

Similar

Tanks and temples: Benchmarking large-scale scene reconstruction
A. Knapitsch, J. Park, Q.-Y. Zhou, and V. Koltun · 2017
Cited alongside, same era.
Working hard to know your neighbor’s margins: Local descriptor learning loss
Original
A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas · 2017
Cited alongside, same era.
Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras
R. Mur-Artal and J. D. Tardós · 2017
Cited alongside, same era.
Demon: Depth and motion network for learning monocular stereo
B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox · 2017
Cited alongside, same era.
Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks
S. Wang, R. Clark, H. Wen, and N. Trigoni · 2017
Cited alongside, same era.
Codeslam—learning a compact, optimisable representation for dense visual slam
M. Bloesch, J. Czarnowski, R. Clark, S. Leutenegger, and A. J. Davison · 2018
Cited alongside, same era.
Ls-net: Learning to solve nonlinear least squares for monocular stereo
Original
R. Clark, M. Bloesch, J. Czarnowski, S. Leutenegger, and A. J. Davison · 2018
Cited alongside, same era.
Superpoint: Self-supervised interest point detection and description
D. DeTone, T. Malisiewicz, and A. Rabinovich · 2018
Cited alongside, same era.
Geodesc: Learning local descriptors by integrating geometry constraints
Z. Luo, T. Shen, L. Zhou, S. Zhu, R. Zhang, Y. Yao, T. Fang, and L. Quan · 2018
Cited alongside, same era.
Omnidirectional dso: Direct sparse odometry with fisheye cameras
H. Matsuki, L. von Stumberg, V. Usenko, J. Stückler, and D. Cremers · 2018
Cited alongside, same era.
Lf-net: Learning local features from images
Original
Y. Ono, E. Trulls, P. Fua, and K. M. Yi · 2018
Cited alongside, same era.
Online temporal calibration for monocular visual-inertial systems
T. Qin and S. Shen · 2018
Cited alongside, same era.

Then

Orb-slam3: An accurate open-source library for visual, visual-inertial and multi-map slam
Original
C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós · 2020
Later among the works it cites.
Deepfactors: Real-time probabilistic dense monocular slam
J. Czarnowski, T. Laidlow, R. Clark, and A. J. Davison · 2020
Later among the works it cites.
Multi-view optimization of local feature geometry
M. Dusmanu, J. L. Schonberger, and M. Pollefeys · 2020
Later among the works it cites.
Robust consistent video depth estimation
Original
J. Kopf, X. Rong, and J.-B. Huang · 2020
Later among the works it cites.
gradslam: Dense slam meets automatic differentiation
J. Krishna Murthy, S. Saryazdi, G. Iyer, and L. Paull · 2020
Later among the works it cites.
Consistent video depth estimation
X. Luo, J.-B. Huang, R. Szeliski, K. Matzen, and J. Kopf · 2020
Later among the works it cites.
Voldor: Visual odometry from log-logistic dense optical flow residuals
Z. Min, Y. Yang, and E. Dunn · 2020
Later among the works it cites.
Kimera: an open-source library for real-time metric-semantic localization and mapping
A. Rosinol, M. Abate, Y. Chang, and L. Carlone · 2020
Later among the works it cites.
Superglue: Learning feature matching with graph neural networks
P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich · 2020
Later among the works it cites.
Nodeslam: Neural object descriptors for multi-view shape reconstruction
E. Sucar, K. Wada, and A. Davison · 2020
Later among the works it cites.
Raft: Recurrent all-pairs field transforms for optical flow
Z. Teed and J. Deng · 2020
Later among the works it cites.
Lm-reloc: Levenberg-marquardt based direct visual relocalization
Original
L. von Stumberg, P. Wenzel, N. Yang, and D. Cremers · 2020
Later among the works it cites.
Tartanvo: A generalizable learning-based vo
Original
W. Wang, Y. Hu, and S. Scherer · 2020
Later among the works it cites.
Tartanair: A dataset to push the limits of visual slam
Original
W. Wang, D. Zhu, X. Wang, Y. Hu, Y. Qiu, C. Wang, Y. Hu, A. Kapoor, and S. Scherer · 2020
Later among the works it cites.
D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry
N. Yang, L. v. Stumberg, R. Wang, and D. Cremers · 2020
Later among the works it cites.
Direct sparse mapping
J. Zubizarreta, I. Aguinaga, and J. M. M. Montiel · 2020
Later among the works it cites.
Ov2 slam: A fully online and versatile visual slam for real-time applications
M. Ferrera, A. Eudes, J. Moras, M. Sanfourche, and G. Le Besnerais · 2021
Closest in time.
Back to the feature: Learning robust camera localization from pixels to pose
Original
P.-E. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V. Larsson, M. Pollefeys, V. Lepetit, L. Hammarstrand, F. Kahl, et al · 2021
Closest in time.
imap: Implicit mapping and positioning in real-time
Original
E. Sucar, S. Liu, J. Ortiz, and A. J. Davison · 2021
Closest in time.
Tangent space backpropagation for 3d transformation groups
Z. Teed and J. Deng · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…