2017

Unsupervised Real-Time Control through Variational Empowerment

Karl, Maximilian, Soelch, Maximilian, Becker-Ehmck, Philip et al.

Understand

We introduce a methodology for efficiently computing a lower bound to empowerment, allowing it to be used as an unsupervised cost function for policy learning in real-time control.

Empowerment, being the channel capacity between actions and states, maximises the influence of an agent on its near future.
It has been shown to be a good model of biological behaviour in the absence of an extrinsic goal.
But empowerment is also prohibitively hard to compute, especially in nonlinear continuous spaces.

Built on

Computation of channel capacity and rate-distortion functions
Blahut, R. (1972) · 1972
Earlier work this paper cites.
Learning internal representations by error propagation
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986) · 1986
Earlier work this paper cites.
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · 1990
Earlier work this paper cites.
Curious model-building control systems
Schmidhuber, J. (1991) · 1991
Earlier work this paper cites.
Td-gammon, a self-teaching backgammon program, achieves master-level play
Tesauro, G. (1994) · 1994
Earlier work this paper cites.
A reinforcement learning approach to job-shop scheduling
Zhang, W. and Dietterich, T. G. (1995) · 1995
Earlier work this paper cites.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., and Singh, S. (1999) · 1999
Earlier work this paper cites.
Optimal stopping of markov processes: Hilbert space theory, approximation algorithms, and an application to pricing high-dimensional financial derivatives
Tsitsiklis, J. N. and Van Roy, B. (1999) · 1999
Earlier work this paper cites.
Regression methods for pricing complex american-style options
Tsitsiklis, J. N. and Van Roy, B. (2001) · 2001
Earlier work this paper cites.
The IM algorithm: A variational approach to information maximization
Barber, D. and Agakov, F. V. (2003) · 2003
Earlier work this paper cites.
Cross channel optimized marketing by reinforcement learning
Abe, N., Verma, N., Apte, C., and Schroko, R. (2004) · 2004
Earlier work this paper cites.
Reinforcement learning for long-run average cost
Gosavi, A. (2004) · 2004
Earlier work this paper cites.
Intrinsically motivated reinforcement learning
Singh, S. P., Barto, A. G., and Chentanez, N. (2004) · 2004
Earlier work this paper cites.
Approximate value iteration in the reinforcement learning context. application to electrical power system control
Ernst, D., Glavic, M., Geurts, P., and Wehenkel, L. (2005) · 2005
Earlier work this paper cites.
Empowerment: A universal agent-centric measure of control
Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2005) · 2005
Earlier work this paper cites.

Similar

Bayesian surprise attracts human attention
Itti, L. and Baldi, P. F. (2006) · 2006
Cited alongside, same era.
Reinforcement learning of local shape in the game of go
Silver, D., Sutton, R. S., and Müller, M. (2007) · 2007
Cited alongside, same era.
Keep your options open: An information-based driving principle for sensorimotor systems
Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2008) · 2008
Cited alongside, same era.
How can we define intrinsic motivation?
Oudeyer, P.-Y. and Kaplan, F. (2008) · 2008
Cited alongside, same era.
Learning exercise policies for american options
Li, Y., Szepesvari, C., and Schuurmans, D. (2009) · 2009
Cited alongside, same era.
Formal theory of creativity, fun, and intrinsic motivation (1990–2010)
Schmidhuber, J. (2010) · 2010
Cited alongside, same era.
Empowerment for Continuous Agent-environment Systems
Jung, T., Polani, D., and Stone, P. (2011) · 2011
Cited alongside, same era.
Reinforcement learning in robotics: A survey
Kober, J. and Peters, J. (2012) · 2012
Cited alongside, same era.
Machine learning: a probabilistic perspective
Murphy, K. P. (2012) · 2012
Cited alongside, same era.
Optimal control and estimation
Stengel, R. F. (2012) · 2012
Cited alongside, same era.
Approximation of empowerment in the continuous domain
Salge, C., Glackin, C., and Polani, D. (2013) · 2013
Cited alongside, same era.
Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem
Schmidhuber, J. (2013) · 2013
Cited alongside, same era.

Then

Empowerment–an introduction
Salge, C., Glackin, C., and Polani, D. (2014) · 2014
Later among the works it cites.
Importance weighted autoencoders
Original
Burda, Y., Grosse, R., and Salakhutdinov, R. (2015) · 2015
Later among the works it cites.
Efficient Empowerment
Original
Karl, M., Bayer, J., and van der Smagt, P. (2015) · 2015
Later among the works it cites.
Variational information maximisation for intrinsically motivated reinforcement learning
Mohamed, S. and Rezende, D. J. (2015) · 2015
Later among the works it cites.
Variational inference with normalizing flows
Rezende, D. J. and Mohamed, S. (2015) · 2015
Later among the works it cites.
Tensorflow: Large-scale machine learning on heterogeneous distributed systems
Original
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · 2016
Later among the works it cites.
Unifying count-based exploration and intrinsic motivation
Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016) · 2016
Later among the works it cites.
Openai gym
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016) · 2016
Later among the works it cites.
Variational intrinsic control
Original
Gregor, K., Rezende, D. J., and Wierstra, D. (2016) · 2016
Later among the works it cites.
Vime: Variational information maximizing exploration
Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. (2016) · 2016
Later among the works it cites.
Improving variational autoencoders with inverse autoregressive flow
Kingma, D. P., Salimans, T., Józefowicz, R., Chen, X., Sutskever, I., and Welling, M. (2016) · 2016
Later among the works it cites.
Mastering the game of go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · 2016
Later among the works it cites.
Surprise-based intrinsic motivation for deep reinforcement learning
Original
Achiam, J. and Sastry, S. (2017) · 2017
Closest in time.
Deep variational bayes filters: Unsupervised learning of state space models from raw data
Karl, M., Soelch, M., Bayer, J., and van der Smagt, P. (2017) · 2017
Closest in time.
Intrinsic motivation and automatic curricula via asymmetric self-play
Original
Sukhbaatar, S., Kostrikov, I., Szlam, A., and Fergus, R. (2017) · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…