2017

Grounded Language Learning in a Simulated 3D World

Hermann, Karl Moritz, Hill, Felix, Green, Simon et al.

Understand

We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf.

This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication.
To achieve this in a scalable fashion, agents must be able to relate language to the world and to actions; that is, their understanding of language must be grounded and embodied.
However, learning grounded language is a notoriously challenging problem in artificial intelligence research.

Built on

A review of BF Skinner’s Verbal Behavior
Noam Chomsky · 1959
Earlier work this paper cites.
Word & Object
W. V. O. Quine · 1960
Earlier work this paper cites.
Understanding natural language
Terry Winograd · 1972
Earlier work this paper cites.
The bootstrapping problem in language acquisition
Steven Pinker · 1987
Earlier work this paper cites.
Backpropagation applied to handwritten zip code recognition
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel · 1989
Earlier work this paper cites.
Mental models of the earth: A study of conceptual change in childhood
Stella Vosniadou and William F Brewer · 1992
Earlier work this paper cites.
Evidence for representations of perceptually similar natural categories by 3-month-old and 4-month-old infants
Paul C Quinn, Peter D Eimas, and Stacey L Rosenkrantz · 1993
Earlier work this paper cites.
Grounding Language in Perception , pages 207–227
Jeffrey Mark Siskind · 1995
Earlier work this paper cites.
Naming in young children: A dumb attentional mechanism?
Linda B Smith, Susan S Jones, and Barbara Landau · 1996
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Grounding the lexical semantics of verbs in visual perception using force dynamics and event logic
Jeffrey Mark Siskind · 2001
Earlier work this paper cites.
Learning words from sights and sounds: A computational model
Deb K Roy and Alex P Pentland · 2002
Earlier work this paper cites.
Semantic cognition: A parallel distributed processing approach
Timothy T Rogers and James L McClelland · 2004
Earlier work this paper cites.

Similar

Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars
Luke S. Zettlemoyer and Michael Collins · 2005
Cited alongside, same era.
Learning to sportscast: A test of grounded language acquisition
David L. Chen and Raymond J. Mooney · 2008
Cited alongside, same era.
A theory of the discovery and predication of relational concepts
Leonidas AA Doumas, John E Hummel, and Catherine M Sandhofer · 2008
Cited alongside, same era.
The symbol grounding problem has been solved. so what’s next
Luc Steels · 2008
Cited alongside, same era.
Language learnability and language development , volume 7
Steven Pinker · 2009
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Cited alongside, same era.
Grounded models of semantic representation
Carina Silberer and Mirella Lapata · 2012
Cited alongside, same era.
Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
Tijmen Tieleman and Geoffrey Hinton · 2012
Cited alongside, same era.
Semantic parsing on freebase from question-answer pairs
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang · 2013
Cited alongside, same era.
Grounded language learning from video described with sentences
Haonan Yu and Jeffrey Mark Siskind · 2013
Cited alongside, same era.
Learning spatially-semantic representations from natural language descriptions and scene classifications
Sachithra Hemachandra, Matthew R. Walter, Stefanie Tellex, and Seth Teller · 2014
Cited alongside, same era.
A framework for learning semantic maps from grounded natural language descriptions
Matthew R. Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth Teller · 2014
Cited alongside, same era.

Then

Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Later among the works it cites.
Language understanding for text-based games using deep reinforcement learning
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay · 2015
Later among the works it cites.
Action-conditional video prediction using deep networks in Atari games
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh · 2015
Later among the works it cites.
Learning to interpret natural language commands through human-robot dialog
Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone · 2015
Later among the works it cites.
Order-embeddings of images and language
Original
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun · 2015
Later among the works it cites.
Show, attend and tell: Neural image caption generation with visual attention
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio · 2015
Later among the works it cites.
Deepmind lab
Original
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen · 2016
Later among the works it cites.
Reinforcement learning with unsupervised auxiliary tasks
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu · 2016
Later among the works it cites.
Learning from explanations using sentiment and advice in RL
Samantha Krening, Brent Harrison, Karen M Feigh, Charles Isbell, Mark Riedl, and Andrea Thomaz · 2016
Later among the works it cites.
Asynchronous methods for deep reinforcement learning
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Later among the works it cites.
Learning language games through interaction
S. I. Wang, P. Liang, and C. Manning · 2016
Later among the works it cites.
Accurately and efficiently interpreting human-robot instructions of varying granularities
Original
Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Lawson L. S. Wong, and Stefanie Tellex · 2017
Closest in time.
A deep compositional framework for human-like language acquisition in virtual environment
Original
Haonan Yu, Haichao Zhang, and Wei Xu · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…