2020

Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning

Zintgraf, Luisa, Feng, Leo, Lu, Cong et al.

Understand

To rapidly learn a new task, it is often essential for agents to explore efficiently -- especially when performance matters from the first timestep.

One way to learn such behaviour is via meta-learning.
Many existing methods however rely on dense rewards for meta-training, and can fail catastrophically if the rewards are sparse.
Without a suitable reward signal, the need for exploration during meta-training is exacerbated.

Built on

Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning
Original
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S · 1910
Earlier work this paper cites.
A possibility for implementing curiosity and boredom in model-building neural controllers
Schmidhuber, J · 1991
Earlier work this paper cites.
Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes
Duff, M. O. and Barto, A · 2002
Earlier work this paper cites.
Reinforcement learning with limited reinforcement: Using bayes risk for active learning in pomdps
Doshi, F., Pineau, J., and Roy, N · 2008
Earlier work this paper cites.
Model-based bayesian reinforcement learning in partially observable domains
Poupart, P. and Vlassis, N · 2008
Earlier work this paper cites.
Bayes-adaptive pomdps
Ross, S., Chaib-draa, B., and Pineau, J · 2008
Earlier work this paper cites.
An analysis of model-based interval estimation for markov decision processes
Strehl, A. L. and Littman, M. L · 2008
Earlier work this paper cites.
Learning to explore and exploit in pomdps
Cai, C., Liao, X., and Carin, L · 2009
Earlier work this paper cites.
A bayesian approach for learning and planning in partially observable markov decision processes
Ross, S., Pineau, J., Chaib-draa, B., and Kreitmann, P · 2011
Earlier work this paper cites.
Acting optimally in partially observable stochastic domains
Cassandra, A. R., Kaelbling, L. P., and Littman, M. L · 2013
Earlier work this paper cites.
Auto-encoding variational bayes
Kingma, D. P. and Welling, M · 2014
Earlier work this paper cites.
Bayesian reinforcement learning: A survey
Ghavamzadeh, M., Mannor, S., Pineau, J., Tamar, A., et al · 2015
Earlier work this paper cites.
Incentivizing exploration in reinforcement learning with deep predictive models
Original
Stadie, B. C., Levine, S., and Abbeel, P · 2015
Earlier work this paper cites.
Surprise-based intrinsic motivation for deep reinforcement learning
Achiam, J. and Sastry, S · 2016
Earlier work this paper cites.

Similar

Unifying count-based exploration and intrinsic motivation
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R · 2016
Cited alongside, same era.
RL 2 : Fast reinforcement learning via slow reinforcement learning
Original
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., and Abbeel, P · 2016
Cited alongside, same era.
Learning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M · 2016
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Finn, C., Abbeel, P., and Levine, S · 2017
Cited alongside, same era.
Deep exploration via randomized value functions
Osband, I., Russo, D., Wen, Z., and Van Roy, B · 2017
Cited alongside, same era.
Count-based exploration with neural density models
Ostrovski, G., Bellemare, M. G., van den Oord, A., and Munos, R · 2017
Cited alongside, same era.
Curiosity-driven exploration by self-supervised prediction
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T · 2017
Cited alongside, same era.
# exploration: A study of count-based exploration for deep reinforcement learning
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, O. X., Duan, Y., Schulman, J., DeTurck, F., and Abbeel, P · 2017
Cited alongside, same era.
Meta-reinforcement learning of structured exploration strategies
Gupta, A., Mendonca, R., Liu, Y., Abbeel, P., and Levine, S · 2018
Cited alongside, same era.
Randomized prior functions for deep reinforcement learning
Osband, I., Aslanides, J., and Cassirer, A · 2018
Cited alongside, same era.
Some considerations on learning to explore via meta-reinforcement learning
Stadie, B. C., Yang, G., Houthooft, R., Chen, X., Duan, Y., Wu, Y., Abbeel, P., and Sutskever, I · 2018
Cited alongside, same era.
Decoupling dynamics and reward for transfer learning
Zhang, A., Satija, H., and Pineau, J · 2018
Cited alongside, same era.

Then

Meta-learning of sequential strategies
Original
Ortega, P. A., Wang, J. X., Rowland, M., Genewein, T., Kurth-Nelson, Z., Pascanu, R., Heess, N., Veness, J., Pritzel, A., Sprechmann, P., et al · 2019
Later among the works it cites.
Efficient off-policy meta-reinforcement learning via probabilistic context variables
Rakelly, K., Zhou, A., Quillen, D., Finn, C., and Levine, S · 2019
Later among the works it cites.
Promp: Proximal meta-policy search
Rothfuss, J., Lee, D., Clavera, I., Asfour, T., and Abbeel, P · 2019
Later among the works it cites.
Using instrinsic motivation for exploration in partially observable environments
Yordanov, Y · 2019
Later among the works it cites.
Conservative uncertainty estimation by fitting prior networks
Ciosek, K., Fortuin, V., Tomioka, R., Hofmann, K., and Turner, R · 2020
Closest in time.
Offline meta reinforcement learning
Original
Dorfman, R. and Tamar, A · 2020
Closest in time.
Meta-learning in neural networks: A survey
Original
Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A · 2020
Closest in time.
Learning adaptive exploration strategies in dynamic environments through informed policy regularization
Original
Kamienny, P.-A., Pirotta, M., Lazaric, A., Lavril, T., Usunier, N., and Denoyer, L · 2020
Closest in time.
Explore then execute: Adapting without rewards via factorized meta-reinforcement learning
Original
Liu, E. Z., Raghunathan, A., Liang, P., and Finn, C · 2020
Closest in time.
Meta-trained agents implement bayes-optimal agents
Mikulik, V., Delétang, G., McGrath, T., Genewein, T., Martic, M., Legg, S., and Ortega, P · 2020
Closest in time.
Uncertainty in neural networks: Approximately bayesian ensembling
Pearce, T., Zaki, M., Brintrup, A., Anastassacos, N., and Neely, A · 2020
Closest in time.
Learn to effectively explore in context-based meta-rl
Original
Zhang, J., Wang, J., Hu, H., Chen, Y., Fan, C., and Zhang, C · 2020
Closest in time.
Varibad: A very good method for bayes-adaptive deep rl via meta-learning
Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hofmann, K., and Whiteson, S · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…