2021

MHER: Model-based Hindsight Experience Replay

Yang, Rui, Fang, Meng, Han, Lei et al.

Understand

Solving multi-goal reinforcement learning (RL) problems with sparse rewards is generally challenging.

Existing approaches have utilized goal relabeling on collected experiences to alleviate issues raised from sparse rewards.
However, these methods are still limited in efficiency and cannot make full use of experiences.
In this paper, we propose Model-based Hindsight Experience Replay (MHER), which exploits experiences more efficiently by leveraging environmental dynamics to generate virtual achieved goals.

Built on

Dyna, an integrated architecture for learning, planning, and reacting
Richard S Sutton · 1991
Earlier work this paper cites.
Learning to achieve goals
Leslie Pack Kaelbling · 1993
Earlier work this paper cites.
A comparison of direct and model-based reinforcement learning
Christopher G Atkeson and Juan Carlos Santamaria · 1997
Earlier work this paper cites.
Policy invariance under reward transformations: Theory and application to reward shaping
Andrew Y Ng, Daishi Harada, and Stuart Russell · 1999
Earlier work this paper cites.
Learning predictive state representations
Satinder P Singh, Michael L Littman, Nicholas K Jong, David Pardoe, and Peter Stone · 2003
Earlier work this paper cites.
Curriculum learning
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston · 2009
Earlier work this paper cites.
Better generalization with forecasts
Tom Schaul and Mark B Ring · 2013
Earlier work this paper cites.
Deterministic policy gradient algorithms
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller · 2014
Earlier work this paper cites.
Continuous control with deep reinforcement learning
Original
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Earlier work this paper cites.
Universal value function approximators
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver · 2015
Earlier work this paper cites.
Openai gym
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Earlier work this paper cites.

Similar

Learning to reinforcement learn
Original
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick · 2016
Cited alongside, same era.
Hindsight experience replay
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba · 2017
Cited alongside, same era.
Curiosity-driven exploration by self-supervised prediction
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell · 2017
Cited alongside, same era.
Sample-efficient reinforcement learning with stochastic ensemble value expansion
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee · 2018
Cited alongside, same era.
Model-based value expansion for efficient model-free reinforcement learning
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine · 2018
Cited alongside, same era.
Addressing function approximation error in actor-critic methods
Scott Fujimoto, Herke Hoof, and David Meger · 2018
Cited alongside, same era.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine · 2018
Cited alongside, same era.
Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine · 2018
Cited alongside, same era.
Goal-conditioned imitation learning
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp · 2019
Cited alongside, same era.
Curriculum-guided hindsight experience replay
Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang · 2019
Cited alongside, same era.
Dream to control: Learning behaviors by latent imagination
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi · 2019
Cited alongside, same era.
When to trust your model: Model-based policy optimization
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine · 2019
Cited alongside, same era.

Then

Addressing sample complexity in visual tasks using HER and hallucinatory gans
Himanshu Sahni, Toby Buckley, Pieter Abbeel, and Ilya Kuzovkin · 2019
Later among the works it cites.
Policy continuation with hindsight inverse dynamics
Hao Sun, Zhizhong Li, Xiaotong Liu, Bolei Zhou, and Dahua Lin · 2019
Later among the works it cites.
Maximum entropy-regularized multi-goal reinforcement learning
Rui Zhao, Xudong Sun, and Volker Tresp · 2019
Later among the works it cites.
Plangan: Model-based planning with sparse rewards and multiple goals
Henry Charlesworth and Giovanni Montana · 2020
Later among the works it cites.
Rewriting history with inverse RL: hindsight inference for policy improvement
Ben Eysenbach, Xinyang Geng, Sergey Levine, and Russ R. Salakhutdinov · 2020
Later among the works it cites.
Generalized hindsight for reinforcement learning
Alexander C. Li, Lerrel Pinto, and Pieter Abbeel · 2020
Later among the works it cites.
Maximum entropy gain exploration for long horizon multi-goal reinforcement learning
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba · 2020
Later among the works it cites.
Mastering atari, go, chess and shogi by planning with a learned model
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al · 2020
Later among the works it cites.
Learning to reach goals via iterated supervised learning
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine · 2021
Closest in time.
Efficient continuous control with double actors and regularized critics
Original
Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, and Xiu Li · 2021
Closest in time.
Mapgo: Model-assisted policy optimization for goal-oriented tasks
Original
Menghui Zhu, Minghuan Liu, Jian Shen, Zhicheng Zhang, Sheng Chen, Weinan Zhang, Deheng Ye, Yong Yu, Qiang Fu, and Wei Yang · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…