2019

Learning Exploration Policies for Navigation

Chen, Tao, Gupta, Saurabh, Gupta, Abhinav

Understand

Numerous past works have tackled the problem of task-driven navigation.

But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention.
In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards.
We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms.

Built on

System identification: theory for the user
Lennart Ljung · 1987
Earlier work this paper cites.
A possibility for implementing curiosity and boredom in model-building neural controllers
Jürgen Schmidhuber · 1991
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams · 1992
Earlier work this paper cites.
A frontier-based approach for autonomous exploration
Brian Yamauchi · 1997
Earlier work this paper cites.
Machine learning and data mining; methods and applications
Ryszard S Michalski, Ivan Bratko, and Avan Bratko · 1998
Earlier work this paper cites.
MINERVA: A second-generation museum tour-guide robot
Sebastian Thrun, Maren Bennewitz, Wolfram Burgard, Armin B Cremers, Frank Dellaert, Dieter Fox, Dirk Hahnel, Charles Rosenberg, Nicholas Roy, Jamieson Schulte, et al · 1999
Earlier work this paper cites.
Multiple view geometry in computer vision
Richard Hartley and Andrew Zisserman · 2003
Earlier work this paper cites.
Intrinsically motivated reinforcement learning
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh · 2005
Earlier work this paper cites.
Probabilistic robotics
Sebastian Thrun, Wolfram Burgard, and Dieter Fox · 2005
Earlier work this paper cites.
Planning algorithms
Steven M LaValle · 2006
Earlier work this paper cites.
Behavioral cloning for simulator validation
Robert G Abbott · 2007
Earlier work this paper cites.
Trajectory optimization using reinforcement learning for map exploration
Thomas Kollar and Nicholas Roy · 2008
Earlier work this paper cites.
A survey of robot learning from demonstration
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning · 2009
Earlier work this paper cites.
A bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot
Ruben Martinez-Cantin, Nando de Freitas, Eric Brochu, José Castellanos, and Arnaud Doucet · 2009
Earlier work this paper cites.
On the comparison of uncertainty criteria for active slam
Henry Carrillo, Ian Reid, and José A Castellanos · 2012
Earlier work this paper cites.

Similar

Exploration in model-based reinforcement learning by empirically estimating learning progress
Manuel Lopes, Tobias Lang, Marc Toussaint, and Pierre-Yves Oudeyer · 2012
Cited alongside, same era.
A frontier-void-based approach for autonomous exploration in 3D
Christian Dornhege and Alexander Kleiner · 2013
Cited alongside, same era.
Incentivizing exploration in reinforcement learning with deep predictive models
Original
Bradly C Stadie, Sergey Levine, and Pieter Abbeel · 2015
Cited alongside, same era.
Information-theoretic exploration with bayesian optimization
Shi Bai, Jinkun Wang, Fanfei Chen, and Brendan Englot · 2016
Cited alongside, same era.
Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, and John J Leonard · 2016
Cited alongside, same era.
RL 2 : Fast reinforcement learning via slow reinforcement learning
Original
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel · 2016
Cited alongside, same era.
EX 2 : Exploration with exemplar models for deep reinforcement learning
Justin Fu, John Co-Reyes, and Sergey Levine · 2017
Cited alongside, same era.
Learning to fly by crashing
Dhiraj Gandhi, Lerrel Pinto, and Abhinav Gupta · 2017
Cited alongside, same era.
Cognitive mapping and planning for visual navigation
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik · 2017
Cited alongside, same era.
Learning to navigate in complex environments
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al · 2017
Cited alongside, same era.
Curiosity-driven exploration by self-supervised prediction
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell · 2017
Cited alongside, same era.
CAD 2 RL: Real single-image flight without a single real image
Fereshteh Sadeghi and Sergey Levine · 2017
Cited alongside, same era.

Then

Autonomous reconstruction of unknown indoor scenes guided by time-varying tensor fields
Kai Xu, Lintao Zheng, Zihao Yan, Guohang Yan, Eugene Zhang, Matthias Niessner, Oliver Deussen, Daniel Cohen-Or, and Hui Huang · 2017
Later among the works it cites.
Preparing for the unknown: Learning a universal policy with online system identification
Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk · 2017
Later among the works it cites.
Neural slam: Learning to explore with external memory
Original
Jingwei Zhang, Lei Tai, Joschka Boedecker, Wolfram Burgard, and Ming Liu · 2017
Later among the works it cites.
On evaluation of embodied navigation agents
Original
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Zamir · 2018
Later among the works it cites.
Embodied Question Answering
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra · 2018
Later among the works it cites.
MapNet: An allocentric spatial memory for mapping environments
Joao F Henriques and Andrea Vedaldi · 2018
Later among the works it cites.
Learning to look around: Intelligently exploring unseen environments for unknown tasks
Dinesh Jayaraman and Kristen Grauman · 2018
Later among the works it cites.
A simple neural attentive meta-learner
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel · 2018
Later among the works it cites.
Neural map: Structured memory for deep reinforcement learning
Emilio Parisotto and Ruslan Salakhutdinov · 2018
Later among the works it cites.
Zero-shot visual imitation
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell · 2018
Later among the works it cites.
Semi-parametric topological memory for navigation
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun · 2018
Later among the works it cites.
Building generalizable agents with a realistic and rich 3d environment
Original
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian · 2018
Later among the works it cites.
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi · 2018
Later among the works it cites.
Large-scale study of curiosity-driven learning
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…