2022

The Challenges of Exploration for Offline Reinforcement Learning

Lambert, Nathan, Wulfmeier, Markus, Whitney, William et al.

Understand

Offline Reinforcement Learning (ORL) enablesus to separately study the two interlinked processes of reinforcement learning: collecting informative experience and inferring optimal behaviour.

The second step has been widely studied in the offline setting, but just as critical to data-efficient RL is the collection of informative data.
The task-agnostic setting for data collection, where the task is not known a priori, is of particular interest due to the possibility of collecting a single dataset and using it to solve several downstream tasks as they arise.
We investigate this setting via curiosity-based intrinsic motivation, a family of exploration methods which encourage the agent to explore those states or transitions it has not yet learned to model.

Built on

What is intrinsic motivation? a typology of computational approaches
Oudeyer, P.-Y. and Kaplan, F · 2009
Earlier work this paper cites.
Model predictive control
Camacho, E. F. and Alba, C. B · 2013
Earlier work this paper cites.
Safe and efficient off-policy reinforcement learning
Original
Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. G · 2016
Earlier work this paper cites.
Hindsight experience replay
Original
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W · 2017
Earlier work this paper cites.
Curiosity-driven exploration by self-supervised prediction
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T · 2017
Earlier work this paper cites.
Information theoretic mpc for model-based reinforcement learning
Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., and Theodorou, E. A · 2017
Earlier work this paper cites.
Maximum a posteriori policy optimisation
Original
Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M · 2018
Earlier work this paper cites.
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., and Levine, S · 2018
Earlier work this paper cites.
Diversity is all you need: Learning skills without a reward function
Original
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S · 2018
Earlier work this paper cites.
Plan online, learn offline: Efficient learning and exploration via model-based control
Original
Lowrey, K., Rajeswaran, A., Kakade, S., Todorov, E., and Mordatch, I · 2018
Earlier work this paper cites.
Deepmind control suite
Original
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al · 2018
Earlier work this paper cites.
A framework for data-driven robotics
Original
Cabi, S., Colmenarejo, S. G., Novikov, A., Konyushkova, K., Reed, S. E., Jeong, R., Zolna, K., Aytar, Y., Budden, D., Vecerík, M., Sushkov, O. O., Barker, D., Scholz, J., Denil, M., de Freitas, N., and Wang, Z · 2019
Earlier work this paper cites.
Evaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks
Original
Dasagi, V., Lee, R., Bruce, J., and Leitner, J · 2019
Earlier work this paper cites.
Low-level control of a quadrotor with deep model-based reinforcement learning
Lambert, N. O., Drew, D. S., Yaconelli, J., Levine, S., Calandra, R., and Pister, K. S. J · 2019
Earlier work this paper cites.

Similar

Advantage-weighted regression: Simple and scalable off-policy reinforcement learning
Original
Peng, X. B., Kumar, A., Zhang, G., and Levine, S · 2019
Cited alongside, same era.
Compositional transfer in hierarchical reinforcement learning
Original
Wulfmeier, M., Abdolmaleki, A., Hafner, R., Springenberg, J. T., Neunert, M., Hertweck, T., Lampe, T., Siegel, N., Heess, N., and Riedmiller, M · 2019
Cited alongside, same era.
An optimistic perspective on offline reinforcement learning
Agarwal, R., Schuurmans, D., and Norouzi, M · 2020
Cited alongside, same era.
Rewriting history with inverse rl: Hindsight inference for policy improvement
Original
Eysenbach, B., Geng, X., Levine, S., and Salakhutdinov, R · 2020
Cited alongside, same era.
D4rl: Datasets for deep data-driven reinforcement learning
Original
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S · 2020
Cited alongside, same era.
Rl unplugged: Benchmarks for offline reinforcement learning
Gulcehre, C., Wang, Z., Novikov, A., Le Paine, T., Gomez Colmenarejo, S., Zolna, K., Agarwal, R., Merel, J., Mankowitz, D., Paduraru, C., et al · 2020
Cited alongside, same era.
Fast task inference with variational intrinsic successor features
Original
Hansen, S. S., Dabney, W., Barreto, A., de Wiele, T. V., Warde-Farley, D., and Mnih, V · 2020
Cited alongside, same era.
Reward-free exploration for reinforcement learning
Jin, C., Krishnamurthy, A., Simchowitz, M., and Yu, T · 2020
Cited alongside, same era.
Conservative q-learning for offline reinforcement learning
Original
Kumar, A., Zhou, A., Tucker, G., and Levine, S · 2020
Cited alongside, same era.
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Original
Levine, S., Kumar, A., Tucker, G., and Fu, J · 2020
Cited alongside, same era.
Generalized hindsight for reinforcement learning
Original
Li, A. C., Pinto, L., and Abbeel, P · 2020
Cited alongside, same era.
Critic regularized regression
Wang, Z., Novikov, A., Zolna, K., Merel, J. S., Springenberg, J. T., Reed, S. E., Shahriari, B., Siegel, N., Gulcehre, C., Heess, N., et al · 2020
Cited alongside, same era.

Then

Efficient self-supervised data collection for offline robot learning
Endrawis, S., Leibovich, G., Jacob, G., Novik, G., and Tamar, A · 2021
Later among the works it cites.
A minimalist approach to offline reinforcement learning
Original
Fujimoto, S. and Gu, S. S · 2021
Later among the works it cites.
Is curiosity all you need? on the utility of emergent behaviours from curious exploration
Original
Groth, O., Wulfmeier, M., Vezzani, G., Dasagi, V., Hertweck, T., Hafner, R., Heess, N., and Riedmiller, M · 2021
Later among the works it cites.
Adaptive reward-free exploration
Kaufmann, E., Ménard, P., Domingues, O. D., Jonsson, A., Leurent, E., and Valko, M · 2021
Later among the works it cites.
Urlb: Unsupervised reinforcement learning benchmark
Laskin, M., Yarats, D., Liu, H., Lee, K., Zhan, A., Lu, K., Cang, C., Pinto, L., and Abbeel, P · 2021
Later among the works it cites.
Behavior from the void: Unsupervised active pre-training
Original
Liu, H. and Abbeel, P · 2021
Later among the works it cites.
Disco rl: Distribution-conditioned reinforcement learning for general-purpose policies
Original
Nasiriany, S., Pong, V. H., Nair, A., Khazatsky, A., Berseth, G., and Levine, S · 2021
Later among the works it cites.
Interesting object, curious agent: Learning task-agnostic exploration
Parisi, S., Dean, V., Pathak, D., and Gupta, A · 2021
Later among the works it cites.
Collect & infer-a fresh look at data-efficient reinforcement learning
Riedmiller, M., Springenberg, J. T., Hafner, R., and Heess, N · 2021
Later among the works it cites.
Understanding the effects of dataset characteristics on offline reinforcement learning
Original
Schweighofer, K., Hofmarcher, M., Dinu, M.-C., Renz, P., Bitto-Nemling, A., Patil, V., and Hochreiter, S · 2021
Later among the works it cites.
Decoupled exploration and exploitation policies for sample-efficient reinforcement learning
Original
Whitney, W. F., Bloesch, M., Springenberg, J. T., Abdolmaleki, A., Cho, K., and Riedmiller, M · 2021
Later among the works it cites.
Representation matters: Improving perception and exploration for robotics
Wulfmeier, M., Byravan, A., Hertweck, T., Higgins, I., Gupta, A., Kulkarni, T., Reynolds, M., Teplyashin, D., Hafner, R., Lampe, T., et al · 2021
Later among the works it cites.
Reinforcement learning with prototypical representations
Original
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L · 2021
Later among the works it cites.
Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning, 2022
Yarats, D., Brandfonbrener, D., Liu, H., Laskin, M., Abbeel, P., Lazaric, A., and Pinto, L · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…