2021

A Dataset Perspective on Offline Reinforcement Learning

Schweighofer, Kajetan, Radler, Andreas, Dinu, Marius-Constantin et al.

Understand

The application of Reinforcement Learning (RL) in real world environments can be expensive or risky due to sub-optimal policies during training.

In Offline RL, this problem is avoided since interactions with an environment are prohibited.
Policies are learned from a given dataset, which solely determines their performance.
Despite this fact, how dataset characteristics influence Offline RL algorithms is still hardly investigated.

Built on

Benchmarking batch deep reinforcement learning algorithms
Original
S. Fujimoto, E. Conti, M. Ghavamzadeh, and J. Pineau · 1910
Earlier work this paper cites.
A mathematical theory of communication
C. E. Shannon · 1948
Earlier work this paper cites.
On a statistical estimate for the entropy of a sequence of independent random variables
G. P. Basharin · 1959
Earlier work this paper cites.
The statistical estimation of entropy in the non-parametric case
B. Harris · 1975
Earlier work this paper cites.
Temporal Credit Assignment in Reinforcement Learning
R. S. Sutton · 1984
Earlier work this paper cites.
Efficient training of artificial neural networks for autonomous navigation
D. A. Pomerleau · 1991
Earlier work this paper cites.
Neural network exploration using optimal experiment design
D. Cohn · 1993
Earlier work this paper cites.
Reinforcement driven information acquisition in non-deterministic environments
J. Storck, S. Hochreiter, and J. Schmidhuber · 1995
Earlier work this paper cites.
Learning in the presence of concept drift and hidden contexts
G. Widmer and M. Kubat · 1996
Earlier work this paper cites.
Efficient model-based exploration
M. Wiering and J. Schmidhuber · 1998
Earlier work this paper cites.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning
R. S. Sutton, D. Precup, and S. Singh · 1999
Earlier work this paper cites.
Symmetries and model minimization in markov decision processes
B. Ravindran and A. G. Barto · 2001
Earlier work this paper cites.
Equivalence notions and model minimization in markov decision processes
R. Givan, T. Dean, and M. Greig · 2003
Earlier work this paper cites.
A survey of exploration strategies in reinforcement learning
R. McFarlane · 2003
Earlier work this paper cites.
State abstraction discovery from irrelevant state variables
N. K. Jong and P. Stone · 2005
Earlier work this paper cites.
Towards a unified theory of state abstraction for MDPs
L. Li, T. J. Walsh, and M. L. Littman · 2006
Earlier work this paper cites.
Hyperloglog: The analysis of a near-optimal cardinality estimation algorithm
P. Flajolet, É. Fusy, O. Gandouet, and F. Meunier · 2007
Earlier work this paper cites.
Matplotlib: A 2d graphics environment
J. D. Hunter · 2007
Earlier work this paper cites.
Scalable parallel programming with cuda
J. Nickolls, I. Buck, M. Garland, and K. Skadron · 2008
Earlier work this paper cites.
Python 3 Reference Manual
G. Van Rossum and F. L. Drake · 2009
Earlier work this paper cites.
Formal theory of creativity, fun, and intrinsic motivation (1990–2010)
J. Schmidhuber · 2010
Earlier work this paper cites.
Batch Reinforcement Learning , pages 45–73
S. Lange, T. Gabel, and M. Riedmiller · 2012
Earlier work this paper cites.

Similar

Playing Atari with deep reinforcement learning
Original
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller · 2013
Cited alongside, same era.
A survey on concept drift adaptation
J. Gama, I. Zliobaite, A. Bifet, P. Mykola, and A. Bouchachia · 2014
Cited alongside, same era.
Openai gym
Original
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba · 2016
Cited alongside, same era.
Generative adversarial imitation learning
J. Ho and S. Ermon · 2016
Cited alongside, same era.
Distributional reinforcement learning with quantile regression
Original
W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos · 2017
Cited alongside, same era.
Self-normalizing neural networks, 2017
G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter · 2017
Cited alongside, same era.
Minimalistic gridworld environment for openai gym
M. Chevalier-Boisvert, L. Willems, and S. Pal · 2018
Cited alongside, same era.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine · 2018
Cited alongside, same era.
Analyzing concept drift and shift from sample data
G. I. Webb, L. K. Lee, B. Goethals, and F. Petitjean · 2018
Cited alongside, same era.
An introduction to domain adaptation and transfer learning
Original
M. K. Wouter · 2018
Cited alongside, same era.
A theory of state abstraction for reinforcement learning
D. Abel · 2019
Cited alongside, same era.
RUDDER: return decomposition for delayed rewards
J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochreiter · 2019
Cited alongside, same era.

Then

RL unplugged: Benchmarks for offline reinforcement learning
Original
C. Gulcehre, Z. Wang, A. Novikov, T. Le Paine, S. G. Colmenarejo, K. Zolna, R. Agarwal, J. Merel, D. Mankowitz, C. Paduraru, G. Dulac-Arnold, J. Li, M. Norouzi, M. Hoffman, O. Nachum, G. Tucker, N. Heess, and N. de Freitas · 2020
Later among the works it cites.
Convergence proof for actor-critic methods applied to PPO and RUDDER
Original
M. Holzleitner, L. Gruber, J. A. Arjona-Medina, J. Brandstetter, and S. Hochreiter · 2020
Later among the works it cites.
Towards continual reinforcement learning: A review and perspectives
Original
K. Khetarpal, M. Riemer, I. Rish, and D. Precup · 2020
Later among the works it cites.
Conservative q-learning for offline reinforcement learning
Original
A. Kumar, A. Zhou, G. Tucker, and S. Levine · 2020
Later among the works it cites.
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Original
S. Levine, A. Kumar, G. Tucker, and J. Fu · 2020
Later among the works it cites.
Offline reinforcement learning hands-on
Original
L. Monier, J. Kmec, A. Laterre, T. Pierrot, V. Courgeau, O. Sigaud, and K. Beguir · 2020
Later among the works it cites.
A unifying view of optimism in episodic reinforcement learning
G. Neu and C. Pike-Burke · 2020
Later among the works it cites.
Align-RUDDER: Learning from few demonstrations by reward redistribution
Original
V. P. Patil, M. Hofmarcher, M.-C. Dinu, M. Dorfer, P. M. Blies, J. Brandstetter, J. A. Arjona-Medina, and S. Hochreiter · 2020
Later among the works it cites.
Rl-cyclegan: Reinforcement learning aware simulation-to-real
K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari · 2020
Later among the works it cites.
Mdp homomorphic networks: Group symmetries in reinforcement learning
E. van der Pol, D. Worrall, H. van Hoof, F. Oliehoek, and M. Welling · 2020
Later among the works it cites.
Critic regularized regression
Original
Z. Wang, A. Novikov, K. Zolna, J. T. Springenberg, S. Reed, B. Shahriari, N. Siegel, J. Merel, C. Gulcehre, N. Heess, and N. de Freitas · 2020
Later among the works it cites.
Bdd100k: A diverse driving dataset for heterogeneous multitask learning
Original
F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell · 2020
Later among the works it cites.
D4rl: Datasets for deep data-driven reinforcement learning
Original
J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine · 2021
Closest in time.
A minimalist approach to offline reinforcement learning
S. Fujimoto and S.S. Gu · 2021
Closest in time.
Regularized behavior value estimation
Original
C. Gulcehre, S. Gómez Colmenarejo, Z. Wang, J. Sygnowski, T. Paine, K. Zolna, Y. Chen, M. Hoffman, R. Pascanu, and N. de Freitas · 2021
Closest in time.
Offline reinforcement learning with fisher divergence critic regularization
I. Kostrikov, R. Fergus, J. Tompson, and O. Nachum · 2021
Closest in time.
Collect & infer - a fresh look at data-efficient reinforcement learning
Original
M. A. Riedmiller, J. T. Springenberg, R. Hafner, and N. Heess · 2021
Closest in time.
Modern Hopfield Networks for Return Decomposition for Delayed Rewards
M. Widrich, M. Hofmarcher, V. P. Patil, A. Bitto-Nemling, and S. Hochreiter · 2021
Closest in time.
A theory of abstraction in reinforcement learning
Original
D. Abel · 2022
Closest in time.
XAI and Strategy Extraction via Reward Redistribution , pages 177–205
M.-C. Dinu, M. Hofmarcher, V. P. Patil, M. Dorfer, P. M. Blies, J. Brandstetter, J. A. Arjona-Medina, and S. Hochreiter · 2022
Closest in time.
Comparing model-free and model-based algorithms for offline reinforcement learning
Original
P. Swazinna, S. Udluft, D. Hein, and T. Runkler · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…