2017

Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

Anderson, Peter, Wu, Qi, Teney, Damien et al.

Understand

A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers.

It is a dream that remains stubbornly distant.
However, recent advances in vision and language methods have made incredible progress in closely related areas.
This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering.

Built on

Procedures as a representation for data in a computer program for understanding natural language
T. Winograd · 1971
Earlier work this paper cites.
Long Short-Term Memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
The Design of Everyday Things
D. A. Norman · 2002
Earlier work this paper cites.
Walk the talk: Connecting language, knowledge, and action in route instructions
M. MacMahon, B. Stankiewicz, and B. Kuipers · 2006
Earlier work this paper cites.
Solving deep memory pomdps with recurrent policy gradients
D. Wierstra, A. Foerster, J. Peters, and J. Schmidhuber · 2007
Earlier work this paper cites.
Natural language command of an autonomous micro-air vehicle
A. S. Huang, S. Tellex, A. Bachrach, T. Kollar, D. Roy, and N. Roy · 2010
Earlier work this paper cites.
Toward understanding natural language directions
T. Kollar, S. Tellex, D. Roy, and N. Roy · 2010
Earlier work this paper cites.
Learning to follow navigational directions
A. Vogel and D. Jurafsky · 2010
Earlier work this paper cites.
Learning to interpret natural language navigation instructions from observations
D. L. Chen and R. J. Mooney · 2011
Earlier work this paper cites.
A reduction of imitation learning and structured prediction to no-regret online learning
S. Ross, G. Gordon, and D. Bagnell · 2011
Earlier work this paper cites.
Understanding natural language commands for robotic navigation and mobile manipulation
S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. J. Teller, and N. Roy · 2011
Earlier work this paper cites.
Unbiased look at dataset bias
A. Torralba and A. A. Efros · 2011
Earlier work this paper cites.
Indoor segmentation and support inference from rgbd images
P. K. Nathan Silberman, Derek Hoiem and R. Fergus · 2012
Earlier work this paper cites.
Grounding spatial relations for human-robot interaction
S. Guadarrama, L. Riano, D. Golland, D. Go, Y. Jia, D. Klein, P. Abbeel, T. Darrell, et al · 2013
Earlier work this paper cites.
Caffe: Convolutional architecture for fast feature embedding
Original
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell · 2014
Earlier work this paper cites.
Referit game: Referring to objects in photographs of natural scenes
S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
D. Kingma and J. Ba · 2014
Earlier work this paper cites.
Effective approaches to attention-based neural machine translation
M.-T. Luong, H. Pham, and C. D. Manning · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
I. Sutskever, O. Vinyals, and Q. V. Le · 2014
Earlier work this paper cites.
VQA: Visual question answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh · 2015
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
D. Bahdanau, K. Cho, and Y. Bengio · 2015
Earlier work this paper cites.

Similar

Scheduled sampling for sequence prediction with recurrent neural networks
S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer · 2015
Cited alongside, same era.
Microsoft COCO Captions: Data Collection and Evaluation Server
Original
X. Chen, T.-Y. L. Hao Fang, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Cited alongside, same era.
SUN RGB-D: A rgb-d scene understanding benchmark suite
S. Song, S. P. Lichtenberg, and J. Xiao · 2015
Cited alongside, same era.
Tensorflow: Large-scale machine learning on heterogeneous distributed systems
Original
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al · 2016
Cited alongside, same era.
Deepmind lab
Original
C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, et al · 2016
Cited alongside, same era.
OpenAI gym
Original
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
M. Kempka, M. Wydmuch, G. Runc, J. Toczek, and W. Jaśkowski · 2016
Cited alongside, same era.
Deep successor reinforcement learning
Original
T. D. Kulkarni, A. Saeedi, S. Gautam, and S. J. Gershman · 2016
Cited alongside, same era.
Professor forcing: A new algorithm for training recurrent networks
A. M. Lamb, A. G. A. P. GOYAL, Y. Zhang, S. Zhang, A. C. Courville, and Y. Bengio · 2016
Cited alongside, same era.
Hierarchical question-image co-attention for visual question answering
J. Lu, J. Yang, D. Batra, and D. Parikh · 2016
Cited alongside, same era.

Then

Gated-attention architectures for task-oriented language grounding
Original
D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov · 2017
Closest in time.
Scannet: Richly-annotated 3d reconstructions of indoor scenes
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner · 2017
Closest in time.
Visual dialog
A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. F. Moura, D. Parikh, and D. Batra · 2017
Closest in time.
Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh · 2017
Closest in time.
Cognitive mapping and planning for visual navigation
S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik · 2017
Closest in time.
AI2-THOR: An interactive 3d environment for visual AI
Original
E. Kolve, R. Mottaghi, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi · 2017
Closest in time.
Press release, October 2017
Matterport · 2017
Closest in time.
Parlai: A dialog research software platform
Original
A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston · 2017
Closest in time.
Learning to navigate in complex environments
P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, et al · 2017
Closest in time.
Mapping instructions and visual observations to actions with reinforcement learning
D. K. Misra, J. Langford, and Y. Artzi · 2017
Closest in time.
Self-critical sequence training for image captioning
S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel · 2017
Closest in time.
MINOS: Multimodal indoor simulator for navigation in complex environments
Original
M. Savva, A. X. Chang, A. Dosovitskiy, T. Funkhouser, and V. Koltun · 2017
Closest in time.
A deep hierarchical approach to lifelong learning in minecraft
C. Tessler, S. Givony, T. Zahavy, D. J. Mankowitz, and S. Mannor · 2017
Closest in time.
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi · 2017
Closest in time.
Bottom-up and top-down attention for image captioning and visual question answering
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang · 2018
Closest in time.
Embodied Question Answering
A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra · 2018
Closest in time.
IQA: Visual question answering in interactive environments
D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi · 2018
Closest in time.
Building generalizable agents with a realistic and rich 3d environment
Original
Y. Wu, Y. Wu, G. Gkioxari, and Y. Tian · 2018
Closest in time.
CHALET: Cornell house agent learning environment
Original
C. Yan, D. Misra, A. Bennnett, A. Walsman, Y. Bisk, and Y. Artzi · 2018
Closest in time.
Gibson Env: Real-world perception for embodied agents
A. R. Zamir, F. Xia, J. He, S. Sax, J. Malik, and S. Savarese · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…