2020

Zero-Shot Compositional Policy Learning via Language Grounding

Cao, Tianshi, Wang, Jingkang, Zhang, Yining et al.

Understand

Despite recent breakthroughs in reinforcement learning (RL) and imitation learning (IL), existing algorithms fail to generalize beyond the training environments.

In reality, humans can adapt to new tasks quickly by leveraging prior knowledge about the world such as language descriptions.
To facilitate the research on language-guided agents with domain adaption, we propose a novel zero-shot compositional policy learning task, where the environments are characterized as a composition of different attributes.
Since there are no public environments supporting this study, we introduce a new research platform BabyAI++ in which the dynamics of environments are disentangled from visual appearance.

Built on

Natural language command of an autonomous micro-air vehicle
A. S. Huang, S. Tellex, A. Bachrach, T. Kollar, D. Roy, and N. Roy · 2010
Earlier work this paper cites.
Toward understanding natural language directions
Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy · 2010
Earlier work this paper cites.
A reduction of imitation learning and structured prediction to no-regret online learning
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell · 2011
Earlier work this paper cites.
Learning to win by reading manuals in a monte-carlo framework
SRK Branavan, David Silver, and Regina Barzilay · 2012
Earlier work this paper cites.
Grounding spatial relations for human-robot interaction
S. Guadarrama, L. Riano, D. Golland, D. Go hring, Y. Jia, D. Klein, P. Abbeel, and T. Darrell · 2013
Earlier work this paper cites.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
Original
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio · 2014
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Earlier work this paper cites.
Deepnav: Learning to navigate large cities
Samarth Brahmbhatt and James Hays · 2017
Earlier work this paper cites.
Home: A household multimodal environment
Original
Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville · 2017
Earlier work this paper cites.
Black-box data-efficient policy search for robotics
Konstantinos I. Chatzilygeroudis, Roberto Rama, Rituraj Kaushik, Dorian Goepp, Vassilis Vassiliades, and Jean-Baptiste Mouret · 2017
Earlier work this paper cites.
Model-agnostic meta-learning for fast adaptation of deep networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Earlier work this paper cites.
Grounded language learning in a simulated 3d world
Original
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al · 2017
Earlier work this paper cites.
AI2-THOR: An Interactive 3D Environment for Visual AI
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi · 2017
Earlier work this paper cites.
A simple neural attentive meta-learner
Original
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel · 2017
Earlier work this paper cites.
Grounding language for transfer in deep reinforcement learning
Original
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola · 2017
Earlier work this paper cites.
Zero-shot task generalization with multi-task deep reinforcement learning
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli · 2017
Earlier work this paper cites.

Similar

Learning to compose skills
Original
Himanshu Sahni, Saurabh Kumar, Farhan Tejani, and Charles Isbell · 2017
Cited alongside, same era.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Cited alongside, same era.
Mastering the game of go without human knowledge
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis · 2017
Cited alongside, same era.
Distral: Robust multitask reinforcement learning
Yee Whye Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu · 2017
Cited alongside, same era.
Multi-task deep reinforcement learning for continuous action control
Zhaoyang Yang, Kathryn E Merrick, Hussein A Abbass, and Lianwen Jin · 2017
Cited alongside, same era.
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel · 2018
Cited alongside, same era.
Gated-attention architectures for task-oriented language grounding
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov · 2018
Cited alongside, same era.
Minimalistic gridworld environment for openai gym
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal · 2018
Cited alongside, same era.
Iqa: Visual question answering in interactive environments
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi · 2018
Cited alongside, same era.
Learning plannable representations with causal infogan
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel · 2018
Cited alongside, same era.
Learning to navigate in cities without a map
Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al · 2018
Cited alongside, same era.
Visual reinforcement learning with imagined goals
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine · 2018
Cited alongside, same era.

Then

Hierarchical reinforcement learning for zero-shot generalization with subtask dependencies
Sungryull Sohn, Junhyuk Oh, and Honglak Lee · 2018
Later among the works it cites.
Universal planning networks
Original
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn · 2018
Later among the works it cites.
Nervenet: Learning structured policy with graph neural networks
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler · 2018
Later among the works it cites.
Building generalizable agents with a realistic and rich 3d environment
Original
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian · 2018
Later among the works it cites.
Interactive grounded language acquisition and generalization in a 2d world
Haonan Yu, Haichao Zhang, and Wei Xu · 2018
Later among the works it cites.
Babyai: A platform to study the sample efficiency of grounded language learning
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio · 2019
Later among the works it cites.
Quantifying generalization in reinforcement learning
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman · 2019
Later among the works it cites.
Using natural language for reward shaping in reinforcement learning
Prasoon Goyal, Scott Niekum, and Raymond J. Mooney · 2019
Later among the works it cites.
Learning to follow directions in street view
Original
Karl Moritz Hermann, Mateusz Malinowski, Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, and Raia Hadsell · 2019
Later among the works it cites.
Are you looking? grounding to multiple modalities in vision-and-language navigation
Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko · 2019
Later among the works it cites.
Compositional generalization through meta sequence-to-sequence learning
Brenden M Lake · 2019
Later among the works it cites.
A survey of reinforcement learning informed by natural language
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel · 2019
Later among the works it cites.
Task-driven modular networks for zero-shot compositional learning
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc’Aurelio Ranzato · 2019
Later among the works it cites.
A causal view of compositional zero-shot recognition
Original
Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik · 2020
Closest in time.
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox · 2020
Closest in time.
Learning unseen concepts via hierarchical decomposition and composition
Muli Yang, Cheng Deng, Junchi Yan, Xianglong Liu, and Dacheng Tao · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…