2020

Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences

Brown, Daniel S., Coleman, Russell, Srinivasan, Ravi et al.

Understand

Bayesian reward learning from demonstrations enables rigorous safety and uncertainty analysis when performing imitation learning.

However, Bayesian reward learning methods are typically computationally intractable for complex control problems.
We propose Bayesian Reward Extrapolation (Bayesian REX), a highly efficient Bayesian reward learning algorithm that scales to high-dimensional imitation learning problems by pre-training a low-dimensional feature encoding via self-supervised tasks and then leveraging preferences over demonstrations to perform fast Bayesian inference.
Bayesian REX can learn to play Atari games from demonstrations, without access to the game score and can generate 100,000 samples from the posterior over reward functions in only 5 minutes on a personal laptop.

Built on

Rank analysis of incomplete block designs: I. the method of paired comparisons
Bradley, R. A. and Terry, M. E · 1952
Earlier work this paper cites.
Efficient training of artificial neural networks for autonomous navigation
Pomerleau, D. A · 1991
Earlier work this paper cites.
A practical bayesian framework for backpropagation networks
MacKay, D. J · 1992
Earlier work this paper cites.
On the complexity of solving markov decision problems
Littman, M. L., Dean, T. L., and Kaelbling, L. P · 1995
Earlier work this paper cites.
Value at risk
Jorion, P · 1997
Earlier work this paper cites.
Policy invariance under reward transformations: Theory and application to reward shaping
Ng, A. Y., Harada, D., and Russell, S · 1999
Earlier work this paper cites.
Algorithms for inverse reinforcement learning
Ng, A. Y. and Russell, S. J · 2000
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour, Y · 2000
Earlier work this paper cites.
Apprenticeship learning via inverse reinforcement learning
Abbeel, P. and Ng, A. Y · 2004
Earlier work this paper cites.
Bayesian inverse reinforcement learning
Ramachandran, D. and Amir, E · 2007
Earlier work this paper cites.
Maximum entropy inverse reinforcement learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K · 2008
Earlier work this paper cites.
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., and Browning, B · 2009
Earlier work this paper cites.
Transfer learning for reinforcement learning domains: A survey
Taylor, M. E. and Stone, P · 2009
Earlier work this paper cites.
Preference-based policy learning
Akrour, R., Schoenauer, M., and Sebag, M · 2011
Earlier work this paper cites.
A reduction of imitation learning and structured prediction to no-regret online learning
Ross, S., Gordon, G., and Bagnell, D · 2011
Earlier work this paper cites.
A bayesian approach for policy learning from trajectory preference queries
Wilson, A., Fern, A., and Tadepalli, P · 2012
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
New learning methods for supervised and unsupervised preference aggregation
Volkovs, M. N. and Zemel, R. S · 2014
Earlier work this paper cites.
Risk-sensitive and robust decision-making: a cvar optimization approach
Chow, Y., Tamar, A., Mannor, S., and Pavone, M · 2015
Earlier work this paper cites.
A comprehensive survey on safe reinforcement learning
Garcıa, J. and Fernández, F · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al · 2015
Earlier work this paper cites.
Action-conditional video prediction using deep networks in atari games
Oh, J., Guo, X., Lee, H., Lewis, R. L., and Singh, S · 2015
Earlier work this paper cites.
Optimizing the cvar via sampling
Tamar, A., Glassner, Y., and Mannor, S · 2015
Earlier work this paper cites.
High-confidence off-policy evaluation
Thomas, P. S., Theocharous, G., and Ghavamzadeh, M · 2015
Earlier work this paper cites.
Concrete problems in ai safety
Original
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D · 2016
Earlier work this paper cites.

Similar

Tutorial on variational autoencoders
Original
Doersch, C · 2016
Cited alongside, same era.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y. and Ghahramani, Z · 2016
Cited alongside, same era.
Safe policy improvement by minimizing robust baseline regret
Ghavamzadeh, M., Petrik, M., and Chow, Y · 2016
Cited alongside, same era.
Cooperative inverse reinforcement learning
Hadfield-Menell, D., Russell, S. J., Abbeel, P., and Dragan, A · 2016
Cited alongside, same era.
Generative adversarial imitation learning
Ho, J. and Ermon, S · 2016
Cited alongside, same era.
Stein variational gradient descent: A general purpose bayesian inference algorithm
Liu, Q. and Wang, D · 2016
Cited alongside, same era.
Risk versus uncertainty in deep learning: Bayes, bootstrap and the dangers of dropout
Osband, I · 2016
Cited alongside, same era.
Successor features for transfer in reinforcement learning
Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D · 2017
Cited alongside, same era.
Deep reinforcement learning from human preferences
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D · 2017
Cited alongside, same era.
Openai baselines
Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y., and Zhokhov, P · 2017
Cited alongside, same era.
Learning robust rewards with adversarial inverse reinforcement learning
Original
Fu, J., Luo, K., and Levine, S · 2017
Cited alongside, same era.
Inverse reward design
Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A · 2017
Cited alongside, same era.

Then

Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning
Brown, D. S. and Niekum, S · 2018
Later among the works it cites.
Risk-aware active inverse reinforcement learning
Brown, D. S., Cui, Y., and Niekum, S · 2018
Later among the works it cites.
Active reward learning from critiques
Cui, Y. and Niekum, S · 2018
Later among the works it cites.
Impossibility and uncertainty theorems in ai value alignment (or why your agi should not have a utility function)
Original
Eckersley, P · 2018
Later among the works it cites.
Rainbow: Combining improvements in deep reinforcement learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D · 2018
Later among the works it cites.
Variational bayesian dropout: Pitfalls and fixes
Hron, J., Matthews, D. G., Ghahramani, Z., et al · 2018
Later among the works it cites.
Learning safe policies with expert guidance
Huang, J., Wu, F., Precup, D., and Cai, Y · 2018
Later among the works it cites.
Reward learning from human preferences and demonstrations in atari
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D · 2018
Later among the works it cites.
Fast and scalable bayesian deep learning by weight-perturbation in adam
Original
Khan, M. E., Nielsen, D., Tangkaratt, V., Lin, W., Gal, Y., and Srivastava, A · 2018
Later among the works it cites.
Projected bnns: Avoiding weight-space pathologies by learning latent representations of neural network weights
Original
Pradier, M. F., Pan, W., Yao, J., Ghosh, S., and Doshi-Velez, F · 2018
Later among the works it cites.
Behavioral cloning from observation
Torabi, F., Warnell, G., and Stone, P · 2018
Later among the works it cites.
Asking easy questions: A user-friendly approach to active reward learning
Bıyık, E., Palan, M., Landolfi, N. C., Losey, D. P., and Sadigh, D · 2019
Later among the works it cites.
Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations
Brown, D. S., Goo, W., Prabhat, N., and Niekum, S · 2019
Later among the works it cites.
Causal confusion in imitation learning
de Haan, P., Jayaraman, D., and Levine, S · 2019
Later among the works it cites.
A divergence minimization perspective on imitation learning methods
Original
Ghasemipour, S. K. S., Zemel, R., and Gu, S · 2019
Later among the works it cites.
Importance sampling policy evaluation with an estimated behavior policy
Hanna, J., Niekum, S., and Stone, P · 2019
Later among the works it cites.
Learning from a learner
Jacq, A., Geist, M., Paiva, A., and Pietquin, O · 2019
Later among the works it cites.
Risk-sensitive generative adversarial imitation learning
Lacotte, J., Ghavamzadeh, M., Chow, Y., and Pavone, M · 2019
Later among the works it cites.
A simple baseline for bayesian uncertainty in deep learning
Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G · 2019
Later among the works it cites.
Ensembledagger: A bayesian approach to safe imitation learning
Menda, K., Driggs-Campbell, K., and Kochenderfer, M. J · 2019
Later among the works it cites.
Learning reward functions by integrating human demonstrations and preferences
Palan, M., Landolfi, N. C., Shevchuk, G., and Sadigh, D · 2019
Later among the works it cites.
Beyond confidence regions: Tight bayesian ambiguity sets for robust mdps
Original
Petrik, M. and Russell, R. H · 2019
Later among the works it cites.
Functional variational bayesian neural networks
Original
Sun, S., Zhang, G., Shi, J., and Grosse, R · 2019
Later among the works it cites.
Safety augmented value estimation from demonstrations (saved): Safe deep model-based rl for sparse cost robotic tasks
Original
Thananjeyan, B., Balakrishna, A., Rosolia, U., Li, F., McAllister, R., Gonzalez, J. E., Levine, S., Borrelli, F., and Goldberg, K · 2019
Later among the works it cites.
Pragmatic-pedagogic value alignment
Fisac, J. F., Gates, M. A., Hamrick, J. B., Liu, C., Hadfield-Menell, D., Palaniappan, M., Malik, D., Sastry, S. S., Griffiths, T. L., and Dragan, A. D · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…