2020

Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning

Chen, Valerie, Gupta, Abhinav, Marino, Kenneth

Understand

Complex, multi-task problems have proven to be difficult to solve efficiently in a sparse-reward reinforcement learning setting.

In order to be sample efficient, multi-task learning requires reuse and sharing of low-level policies.
To facilitate the automatic decomposition of hierarchical tasks, we propose the use of step-by-step human demonstrations in the form of natural language instructions and action trajectories.
We introduce a dataset of such demonstrations in a crafting-based grid world.

Built on

Intrinsic motivation and its role in psychological development
JMcVLevine Hunt · 1965
Earlier work this paper cites.
Imitation, objects, tools, and the rudiments of language in human ontogeny
Andrew N Meltzoff · 1988
Earlier work this paper cites.
Feudal reinforcement learning
Peter Dayan and Geoffrey E Hinton · 1993
Earlier work this paper cites.
Reinforcement learning with hierarchies of machines
Ronald Parr and Stuart J Russell · 1998
Earlier work this paper cites.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning
Richard S Sutton, Doina Precup, and Satinder Singh · 1999
Earlier work this paper cites.
How people learn: Brain, mind, experience, and school: Expanded edition
National Research Council et al · 2000
Earlier work this paper cites.
Learning options in reinforcement learning
Martin Stolle and Doina Precup · 2002
Earlier work this paper cites.
Learning to interpret natural language navigation instructions from observations
David L Chen and Raymond J Mooney · 2011
Earlier work this paper cites.
Understanding natural language commands for robotic navigation and mobile manipulation
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy · 2011
Earlier work this paper cites.
Learning to win by reading manuals in a monte-carlo framework
SRK Branavan, David Silver, and Regina Barzilay · 2012
Earlier work this paper cites.
Learning and transfer of modulated locomotor controllers
Original
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver · 2016
Earlier work this paper cites.
Listen, attend, and walk: Neural mapping of navigational instructions to action sequences
Hongyuan Mei, Mohit Bansal, and Matthew R Walter · 2016
Earlier work this paper cites.
Modular multitask reinforcement learning with policy sketches
Jacob Andreas, Dan Klein, and Sergey Levine · 2017
Earlier work this paper cites.
Stochastic neural networks for hierarchical reinforcement learning
Carlos Florensa, Yan Duan, and Pieter Abbeel · 2017
Earlier work this paper cites.

Similar

Intrinsically motivated goal exploration processes with automatic curriculum learning
Original
Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Yves Oudeyer · 2017
Cited alongside, same era.
Zero-shot task generalization with multi-task deep reinforcement learning
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli · 2017
Cited alongside, same era.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Cited alongside, same era.
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel · 2018
Cited alongside, same era.
Learning with latent language
Jacob Andreas, Dan Klein, and Sergey Levine · 2018
Cited alongside, same era.
Gated-attention architectures for task-oriented language grounding
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov · 2018
Cited alongside, same era.
Guiding policies with language via meta-learning
John D Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, Jacob Andreas, John DeNero, Pieter Abbeel, and Sergey Levine · 2018
Cited alongside, same era.
Embodied Question Answering
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra · 2018
Cited alongside, same era.
Diversity is all you need: Learning skills without a reward function
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine · 2018
Cited alongside, same era.
Automatic goal generation for reinforcement learning agents
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel · 2018
Cited alongside, same era.
Learning an embedding space for transferable robot skills
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller · 2018
Cited alongside, same era.
Hierarchical imitation and reinforcement learning
Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé · 2018
Cited alongside, same era.

Then

Hierarchical and interpretable skill acquisition in multi-task reinforcement learning
Tianmin Shu, Caiming Xiong, and Richard Socher · 2018
Later among the works it cites.
Learning to parse natural language to grounded reward functions with weak supervision
Edward C Williams, Nakul Gopalan, Mine Rhee, and Stefanie Tellex · 2018
Later among the works it cites.
Interactive grounded language acquisition and generalization in a 2d world
Haonan Yu, Haichao Zhang, and Wei Xu · 2018
Later among the works it cites.
Self-educated language agent with hindsight experience replay for instruction following
Original
Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin · 2019
Later among the works it cites.
Hierarchical decision making by generating and following natural language instructions
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis · 2019
Later among the works it cites.
Language as an abstraction for hierarchical deep reinforcement learning
Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn · 2019
Later among the works it cites.
A survey of reinforcement learning informed by natural language
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel · 2019
Later among the works it cites.
Hierarchical rl using an ensemble of proprioceptive periodic policies
Kenneth Marino, Abhinav Gupta, Rob Fergus, and Arthur Szlam · 2019
Later among the works it cites.
Word2vec to behavior: morphology facilitates the grounding of language in machines
David Matthews, Sam Kriegman, Collin Cappelle, and Josh Bongard · 2019
Later among the works it cites.
Vision-based navigation with language-based assistance via imitation learning with indirect intervention
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan · 2019
Later among the works it cites.
Babyai++: Towards grounded-language learning beyond memorization
Original
Tianshi Cao, Jingkang Wang, Yining Zhang, and Sivabalan Manivasagam · 2020
Closest in time.
Grounding language in play
Original
Corey Lynch and Pierre Sermanet · 2020
Closest in time.
Rtfm: Generalising to new environment dynamics via reading
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…