2018

FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning

Shah, Pararth, Fiser, Marek, Faust, Aleksandra et al.

Understand

Understanding and following directions provided by humans can enable robots to navigate effectively in unknown situations.

We present FollowNet, an end-to-end differentiable neural architecture for learning multi-modal navigation policies.
FollowNet maps natural language instructions as well as visual and depth inputs to locomotion primitives.
FollowNet processes instructions using an attention mechanism conditioned on its visual and depth input to focus on the relevant parts of the command while performing the navigation task.

Built on

Understanding natural language commands for robotic navigation and mobile manipulation
S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. Teller, and N. Roy · 2011
Earlier work this paper cites.
Understanding natural language commands for robotic navigation and mobile manipulation
S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. J. Teller, and N. Roy · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Game Theory and Multi-agent Reinforcement Learning
A. Nowé, P. Vrancx, and Y.-M. Hauwere · 2012
Earlier work this paper cites.
Learning to Parse Natural Language Commands to a Robot Control System
C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean · 2013
Earlier work this paper cites.
Learning phrase representations using rnn encoder–decoder for statistical machine translation
K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio · 2014
Earlier work this paper cites.
Efficient motion-based task learning for a serial link manipulator
N. Malone, A. Faust, B. Rohrer, R. Lumia, J. Wood, and L. Tapia · 2014
Earlier work this paper cites.
Using recurrent neural networks for slot filling in spoken language understanding
G. Mesnil, Y. Dauphin, K. Yao, Y. Bengio, L. Deng, D. Hakkani-Tur, X. He, L. Heck, G. Tur, D. Yu, et al · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al · 2015
Earlier work this paper cites.

Similar

Learning to interpret natural language commands through human-robot dialog
J. Thomason, S. Zhang, R. Mooney, and P. Stone · 2015
Cited alongside, same era.
Learning to interpret natural language commands through human-robot dialog
J. Thomason, S. Zhang, R. J. Mooney, and P. Stone · 2015
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Listen, attend, and walk: Neural mapping of navigational instructions to action sequences
H. Mei, M. Bansal, and M. R. Walter · 2016
Cited alongside, same era.
Value iteration networks
A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel · 2016
Cited alongside, same era.
Learning multi-modal grounded linguistic semantics by playing ”i spy”
J. Thomason, J. Sinapov, M. Svetlik, P. Stone, and R. J. Mooney · 2016
Cited alongside, same era.
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
Original
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. D. Reid, S. Gould, and A. van den Hengel · 2017
Cited alongside, same era.
Accurately and efficiently interpreting human-robot instructions of varying granularities
D. Arumugam, S. Karamcheti, N. Gopalan, L. L. S. Wong, and S. Tellex · 2017
Cited alongside, same era.
Google Vizier: A Service for Black-Box Optimization
D. Golovin, B. Solnik, S. Moitra, G. Kochanski, J. E. Karro, and D. Sculley, editors · 2017
Cited alongside, same era.
Grounded language learning in a simulated 3d world
Original
K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czarnecki, M. Jaderberg, D. Teplyashin, M. Wainwright, C. Apps, D. Hassabis, and P. Blunsom · 2017
Cited alongside, same era.

Then

Mapping instructions and visual observations to actions with reinforcement learning
D. Misra, J. Langford, and Y. Artzi · 2017
Later among the works it cites.
Coalition formation for multi-agent pursuit based on neural network and AGRMF model
Original
Z. Pei, S. Piao, and M. Souidi · 2017
Later among the works it cites.
Semantic scene completion from a single depth image
S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser · 2017
Later among the works it cites.
Multi-modal word synset induction
J. Thomason and R. J. Mooney · 2017
Later among the works it cites.
Opportunistic active learning for grounding natural language descriptions
J. Thomason, A. Padmakumar, J. Sinapov, J. Hart, P. Stone, and R. J. Mooney · 2017
Later among the works it cites.
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi · 2017
Later among the works it cites.
Gated-attention architectures for task-oriented language grounding
D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov · 2018
Closest in time.
Embodied question answering
A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra · 2018
Closest in time.
PRM-RL: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning
A. Faust, O. Ramirez, M. Fiser, K. Oslund, A. Francis, J. Davidson, and L. Tapia · 2018
Closest in time.
Interactive grounded language acquisition and generalization in a 2d world
H. Yu, H. Zhang, and W. Xu · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…