2019

Self-Supervised Exploration via Disagreement

Pathak, Deepak, Gandhi, Dhiraj, Gupta, Abhinav

Understand

Efficient exploration is a long-standing problem in sensorimotor learning.

Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation.
However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups.
In this paper, we propose a formulation for exploration inspired by the work in active learning literature.

Built on

Learning synergies between pushing and grasping with self-supervised deep reinforcement learning
Original
Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., and Funkhouser, T. A · 1906
Earlier work this paper cites.
Curious model-building control systems
Schmidhuber, J · 1991
Earlier work this paper cites.
Query by committee
Seung, H., Opper, M., and Sompolinsky, H · 1992
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J · 1992
Earlier work this paper cites.
A sequential algorithm for training text classifiers
Lewis, D. and Gale, W · 1994
Earlier work this paper cites.
Committee-based sampling for training probabilistic classifiers
Dagan, I. and Engelson, S · 1995
Earlier work this paper cites.
Convex optimization
Boyd, S. and Vandenberghe, L · 2004
Earlier work this paper cites.
An analytic solution to discrete bayesian reinforcement learning
Poupart, P., Vlassis, N., Hoey, J., and Regan, K · 2006
Earlier work this paper cites.
Intrinsic motivation systems for autonomous mental development
Oudeyer, P.-Y., Kaplan, F., and Hafner, V. V · 2007
Earlier work this paper cites.
An analysis of model-based interval estimation for markov decision processes
Strehl, A. and Littman, M · 2008
Earlier work this paper cites.
Near-bayesian exploration in polynomial time
Kolter, Z. and Ng, A · 2009
Earlier work this paper cites.
What is intrinsic motivation? a typology of computational approaches
Oudeyer, P.-Y. and Kaplan, F · 2009
Earlier work this paper cites.
Active learning literature survey
Settles, B · 2010
Earlier work this paper cites.
Bayesian active learning for classification and preference learning
Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M · 2011
Earlier work this paper cites.

Similar

Planning to be surprised: Optimal bayesian exploration in dynamic environments
Sun, Y., Gomez, F., and Schmidhuber, J · 2011
Cited alongside, same era.
Exploration in model-based reinforcement learning by empirically estimating learning progress
Lopes, M., Lang, T., Toussaint, M., and Oudeyer, P.-Y · 2012
Cited alongside, same era.
An information-theoretic approach to curiosity-driven reinforcement learning
Still, S. and Precup, D · 2012
Cited alongside, same era.
Estimating or propagating gradients through stochastic neurons for conditional computation
Original
Bengio, Y., Léonard, N., and Courville, A · 2013
Cited alongside, same era.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Original
Gal, Y. and Ghahramani, Z · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D · 2015
Cited alongside, same era.
Unifying count-based exploration and intrinsic motivation
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R · 2016
Cited alongside, same era.
Improving pilco with bayesian neural network dynamics models
Gal, Y., McAllister, R., and Rasmussen, C. E · 2016
Cited alongside, same era.
Vime: Variational information maximizing exploration
Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P · 2016
Cited alongside, same era.
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D · 2016
Cited alongside, same era.
Deep exploration via bootstrapped dqn
Osband, I., Blundell, C., Pritzel, A., and Van Roy, B · 2016
Cited alongside, same era.
Surprise-based intrinsic motivation for deep reinforcement learning
Original
Achiam, J. and Sastry, S · 2017
Cited alongside, same era.

Then

Ex2: Exploration with exemplar models for deep reinforcement learning
Fu, J., Co-Reyes, J. D., and Levine, S · 2017
Later among the works it cites.
Variational intrinsic control
Gregor, K., Rezende, D. J., and Wierstra, D · 2017
Later among the works it cites.
Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents
Original
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M. J., and Bowling, M · 2017
Later among the works it cites.
Curiosity-driven exploration by self-supervised prediction
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T · 2017
Later among the works it cites.
Parameter space noise for exploration
Original
Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychowicz, M · 2017
Later among the works it cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Later among the works it cites.
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Original
Chua, K., Calandra, R., McAllister, R., and Levine, S · 2018
Later among the works it cites.
Diversity is all you need: Learning skills without a reward function
Original
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S · 2018
Later among the works it cites.
Unity: A general platform for intelligent agents
Original
Juliani, A., Berges, V.-P., Vckay, E., Gao, Y., Henry, H., Mattar, M., and Lange, D · 2018
Later among the works it cites.
Intrinsic motivation and automatic curricula via asymmetric self-play
Sukhbaatar, S., Kostrikov, I., Szlam, A., and Fergus, R · 2018
Later among the works it cites.
Large-scale study of curiosity-driven learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A · 2019
Closest in time.
Model-predictive policy learning with uncertainty regularization for driving in dense traffic
Henaff, M., Canziani, A., and LeCun, Y · 2019
Closest in time.
Model-Based Active Exploration
Shyam, P., Jaśkowski, W., and Gomez, F · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…