2019

HIGhER : Improving instruction following with Hindsight Generation for Experience Replay

Cideron, Geoffrey, Seurin, Mathieu, Strub, Florian et al.

Understand

Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality.

While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language understanding and reinforcement learning in even simple instruction following scenarios.
This joint learning problem is alleviated through expert demonstrations, auxiliary losses, or neural inductive biases.
In this paper, we propose an orthogonal approach called Hindsight Generation for Experience Replay (HIGhER) that extends the Hindsight Experience Replay (HER) approach to the language-conditioned policy setting.

Built on

“Policy invariance under reward transformations: Theory and application to reward shaping”
Andrew Ng, Daishi Harada and Stuart Russell · 1999
Earlier work this paper cites.
“Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text”, 2020
Original
Felix Hill, Sona Mokra, Nathaniel Wong and Tim Harley · 2005
Earlier work this paper cites.
“Learning to interpret natural language navigation instructions from observations”
David Chen and Raymond Mooney · 2011
Earlier work this paper cites.
“Understanding natural language commands for robotic navigation and mobile manipulation”
Stefanie Tellex et al · 2011
Earlier work this paper cites.
“Weakly supervised learning of semantic parsers for mapping instructions to actions”
Yoav Artzi and Luke Zettlemoyer · 2013
Earlier work this paper cites.
“Compression and communication in the cultural evolution of linguistic structure”
Simon Kirby, Monica Tamariz, Hannah Cornish and Kenny Smith · 2015
Earlier work this paper cites.
“Human-level control through deep reinforcement learning”
Volodymyr Mnih et al · 2015
Earlier work this paper cites.
“Universal value function approximators”
Tom Schaul, Daniel Horgan, Karol Gregor and David Silver · 2015
Earlier work this paper cites.
“Neural module networks”
Jacob Andreas, Marcus Rohrbach, Trevor Darrell and Dan Klein · 2016
Earlier work this paper cites.
“Unifying count-based exploration and intrinsic motivation”
Marc Bellemare et al · 2016
Earlier work this paper cites.
“Deep reinforcement learning with double Q-Learning”
Hado Hasselt, Arthur Guez and David Silver · 2016
Earlier work this paper cites.
“Virtual embodiment: A scalable long-term strategy for artificial intelligence research”
Original
Douwe Kiela, Luana Bulat, Anita Vero and Stephen Clark · 2016
Earlier work this paper cites.
“Prioritized experience replay”
Tom Schaul, John Quan, Ioannis Antonoglou and David Silver · 2016
Earlier work this paper cites.
“Dueling Network Architectures for Deep Reinforcement Learning”
Ziyu Wang et al · 2016
Earlier work this paper cites.

Similar

“Hindsight experience replay”
Marcin Andrychowicz et al · 2017
Cited alongside, same era.
“HoME: a Household Multimodal Environment”
Simon Brodeur et al · 2017
Cited alongside, same era.
“Grounded language learning in a simulated 3d world”
Original
Karl Hermann et al · 2017
Cited alongside, same era.
“Understanding grounded language learning agents”
Original
Felix Hill, Karl Hermann, Phil Blunsom and Stephen Clark · 2017
Cited alongside, same era.
“Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments”
Peter Anderson et al · 2018
Cited alongside, same era.
“ACTRCE: Augmenting Experience via Teacher’s Advice For Multi-Goal Reinforcement Learning”
Harris Chan et al · 2018
Cited alongside, same era.
“Gated-attention architectures for task-oriented language grounding”
Devendra Chaplot et al · 2018
Cited alongside, same era.
“Speaker-follower models for vision-and-language navigation”
Daniel Fried et al · 2018
Cited alongside, same era.
“Deep q-learning from demonstrations”
Todd Hester et al · 2018
Cited alongside, same era.
“Compositional Attention Networks for Machine Reasoning”
Drew Hudson and Christopher. Manning · 2018
Cited alongside, same era.
“Grounding language for transfer in deep reinforcement learning”
Karthik Narasimhan, Regina Barzilay and Tommi Jaakkola · 2018
Cited alongside, same era.
“Film: Visual reasoning with a general conditioning layer”
Ethan Perez et al · 2018
Cited alongside, same era.

Then

“Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation”
Xiaoxue Zang et al · 2018
Later among the works it cites.
“Learning to Understand Goal Specifications by Modelling Reward”
Dzmitry Bahdanau et al · 2019
Closest in time.
“Systematic Generalization: What Is Required and Can It Be Learned?”
Dzmitry Bahdanau et al · 2019
Closest in time.
“BabyAI: First Steps Towards Grounded Language Learning With a Human In the Loop”
Maxime Chevalier-Boisvert et al · 2019
Closest in time.
“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”
Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova · 2019
Closest in time.
“DHER: Hindsight Experience Replay for Dynamic Goals”
Meng Fang et al · 2019
Closest in time.
“From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following”
Justin Fu, Anoop Korattikara, Sergey Levine and Sergio Guadarrama · 2019
Closest in time.
“Learning to follow directions in street view”
Original
Karl Hermann et al · 2019
Closest in time.
“Language as an abstraction for hierarchical deep reinforcement learning”
Yiding Jiang, Shixiang Gu, Kevin Murphy and Chelsea Finn · 2019
Closest in time.
“Competitive experience replay”
Hao Liu, Alexander Trott, Richard Socher and Caiming Xiong · 2019
Closest in time.
“A Survey of Reinforcement Learning Informed by Natural Language”
Jelena Luketina et al · 2019
Closest in time.
“Hindsight policy gradients”
Paulo Rauber, Avinash Ummadisingu, Filipe Mutz and Jürgen Schmidhuber · 2019
Closest in time.
“Visual Hindsight Experience Replay”
Himanshu Sahni, Toby Buckley, Pieter Abbeel and Ilya Kuzovkin · 2019
Closest in time.
“Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation”
Xin Wang et al · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…