2024

Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theory

Zhang, Zhi, Chow, Chris, Zhang, Yasi et al.

Understand

Lifelong reinforcement learning (RL) has been developed as a paradigm for extending single-task RL to more realistic, dynamic settings.

In lifelong RL, the "life" of an RL agent is modeled as a stream of tasks drawn from a task distribution.
We propose EPIC (Empirical PAC-Bayes that Improves Continuously), a novel algorithm designed for lifelong RL using PAC-Bayes theory.
EPIC learns a shared policy distribution, referred to as the world policy, which enables rapid adaptation to new tasks while retaining valuable knowledge from previous experiences.

Built on

On tail probabilities for martingales
Freedman, D. A. (1975) · 1975
Earlier work this paper cites.
Asymptotic evaluation of certain markov process expectations for large time. iv
Donsker, M. D. and Varadhan, S. S. (1983) · 1983
Earlier work this paper cites.
Meta-neural networks that learn by learning
Naik, D. K. and Mammone, R. J. (1992) · 1992
Earlier work this paper cites.
Bayesian experimental design: A review
Chaloner, K. and Verdinelli, I. (1995) · 1995
Earlier work this paper cites.
Learning to Learn: Introduction and Overview
Thrun, S. and Pratt, L. (1998) · 1998
Earlier work this paper cites.
Some PAC-Bayesian Theorems
McAllester, D. A. (1999) · 1999
Earlier work this paper cites.
PAC-Bayes & margins
Langford, J. and Shawe-Taylor, J. (2002) · 2002
Earlier work this paper cites.
PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification
Seeger, M. (2002) · 2002
Earlier work this paper cites.
Probably Approximately Correct Vision-Based Planning using Motion Primitives
Original
Veer, S. and Majumdar, A. (2020) · 2002
Earlier work this paper cites.
On the role of data in PAC-Bayes bounds
Original
Dziugaite, G. K., Hsu, K., Gharbieh, W., Arpino, G., and Roy, D. M. (2020) · 2006
Earlier work this paper cites.
PAC-Bayesian learning of linear classifiers
Germain, P., Lacasse, A., Laviolette, F., and Marchand, M. (2009) · 2009
Earlier work this paper cites.
PAC-Bayesian Model Selection for Reinforcement Learning
Fard, M. and Pineau, J. (2010) · 2010
Earlier work this paper cites.
PAC-Bayesian Policy Evaluation for Reinforcement Learning
Original
Fard, M. M., Pineau, J., and Szepesvari, C. (2012) · 2012
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Original
Saxe, A. M., McClelland, J. L., and Ganguli, S. (2014) · 2014
Earlier work this paper cites.
Continuous control with deep reinforcement learning
Original
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015) · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., et al. (2015) · 2015
Earlier work this paper cites.
Trust region policy optimization
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015) · 2015
Earlier work this paper cites.

Similar

Data-dependent Initializations of Convolutional Neural Networks
Original
Krähenbühl, P., Doersch, C., Donahue, J., and Darrell, T. (2016) · 2016
Cited alongside, same era.
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Original
Salimans, T. and Kingma, D. P. (2016) · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · 2016
Cited alongside, same era.
Robust and efficient transfer learning with hidden parameter markov decision processes
Killian, T. W., Daulton, S., Konidaris, G., and Doshi-Velez, F. (2017) · 2017
Cited alongside, same era.
Exploring Generalization in Deep Learning
Original
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017) · 2017
Cited alongside, same era.
Mastering chess and shogi by self-play with a general reinforcement learning algorithm
Original
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017) · 2017
Cited alongside, same era.
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., and Farhadi, A. (2017) · 2017
Cited alongside, same era.
Policy and value transfer in lifelong reinforcement learning
Abel, D., Jinnai, Y., Guo, S. Y., Konidaris, G., and Littman, M. (2018) · 2018
Cited alongside, same era.
Meta-learning by adjusting priors based on extended pac-bayes theory
Amit, R. and Meir, R. (2018) · 2018
Cited alongside, same era.
Planning and learning with stochastic action sets
Original
Boutilier, C., Cohen, A., Daniely, A., Hassidim, A., Mansour, Y., Meshi, O., Mladenov, M., and Schuurmans, D. (2018) · 2018
Cited alongside, same era.
World models
Original
Ha, D. and Schmidhuber, J. (2018) · 2018
Cited alongside, same era.
A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
Original
Neyshabur, B., Bhojanapalli, S., and Srebro, N. (2018) · 2018
Cited alongside, same era.

Then

Regret minimization experience replay in off-policy reinforcement learning
Liu, X.-H., Xue, Z., Pang, J., Jiang, S., Xu, F., and Yu, Y. (2021) · 2021
Later among the works it cites.
Pac-bayes control: learning policies that provably generalize to novel environments
Majumdar, A., Farid, A., and Sonar, A. (2021) · 2021
Later among the works it cites.
Reinforcement learning with sparse-executing actions via sparsity regularization
Original
Pang, J.-C., Xu, T., Jiang, S., Liu, Y.-R., and Yu, Y. (2021) · 2021
Later among the works it cites.
Syndicated bandits: A framework for auto tuning hyper-parameters in contextual bandit algorithms
Ding, Q., Kang, Y., Liu, Y.-W., Lee, T. C. M., Hsieh, C.-J., and Sharpnack, J. (2022) · 2022
Later among the works it cites.
Pac-bayesian lifelong learning for multi-armed bandits
Flynn, H., Reeb, D., Kandemir, M., and Peters, J. (2022) · 2022
Later among the works it cites.
Model-based lifelong reinforcement learning with bayesian exploration
Fu, H., Yu, S., Littman, M., and Konidaris, G. (2022) · 2022
Later among the works it cites.
What can transformers learn in-context? a case study of simple function classes
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. (2022) · 2022
Later among the works it cites.
Efficient frameworks for generalized low-rank matrix bandit problems
Kang, Y., Hsieh, C.-J., and Lee, T. C. M. (2022) · 2022
Later among the works it cites.
Towards continual reinforcement learning: A review and perspectives
Khetarpal, K., Riemer, M., Rish, I., and Precup, D. (2022) · 2022
Later among the works it cites.
Reactive exploration to cope with non-stationarity in lifelong reinforcement learning
Steinparz, C. A., Schmied, T., Paischer, F., Dinu, M.-C., Patil, V. P., Bitto-Nemling, A., Eghbal-zadeh, H., and Hochreiter, S. (2022) · 2022
Later among the works it cites.
A general sample complexity analysis of vanilla policy gradient
Yuan, R., Gower, R. M., and Lazaric, A. (2022) · 2022
Later among the works it cites.
Prediction and control in continual reinforcement learning
Anand, N. and Precup, D. (2023) · 2023
Later among the works it cites.
Robust lipschitz bandits to adversarial corruptions
Kang, Y., Hsieh, C.-J., and Lee, T. C. M. (2023) · 2023
Later among the works it cites.
The effectiveness of world models for continual reinforcement learning
Kessler, S., Ostaszewski, M., Bortkiewicz, M., Żarski, M., Wolczyk, M., Parker-Holder, J., Roberts, S. J., Mi, P., et al. (2023) · 2023
Later among the works it cites.
Statistical guarantees for variational autoencoders using pac-bayesian theory
Mbacke, S. D., Clerc, F., and Germain, P. (2023) · 2023
Later among the works it cites.
A definition of continual reinforcement learning
Abel, D., Barreto, A., Van Roy, B., Precup, D., van Hasselt, H. P., and Singh, S. (2024) · 2024
Closest in time.
Online continuous hyperparameter optimization for generalized linear contextual bandits
Kang, Y., Hsieh, C.-J., and Lee, T. (2024) · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…