2021

DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policies

Nasiriany, Soroush, Pong, Vitchyr H., Nair, Ashvin et al.

Understand

Can we use reinforcement learning to learn general-purpose policies that can perform a wide range of different tasks, resulting in flexible and reusable skills? Contextual policies provide this capability in principle, but the representation of the context determines the degree of generalization and expressivity.

Categorical contexts preclude generalization to entirely new tasks.
Goal-conditioned policies may enable some generalization, but cannot capture all tasks that might be desired.
In this paper, we propose goal distributions as a general and broadly applicable task representation suitable for contextual policies.

Built on

L. P. Kaelbling, “Learning to achieve goals,” in
1993
Earlier work this paper cites.
S. Koenig and R. G. Simmons, “The effect of representation and knowledge on goal-directed exploration with reinforcement-learning algorithms,”
1996
Earlier work this paper cites.
R. S. Sutton and A. G. Barto,
1998
Earlier work this paper cites.
T. P. Minka, “Expectation propagation for approximate bayesian inference,” in
2001
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” in
2014
Earlier work this paper cites.
D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic Backpropagation and Approximate Inference in Deep Generative Models,” in
2014
Earlier work this paper cites.
T. Schaul, D. Horgan, K. Gregor, and D. Silver, “Universal Value Function Approximators,” in
2015
Earlier work this paper cites.
A. Hallak, D. Di Castro, and S. Mannor, “Contextual markov decision processes,”
Original
2015
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver
2015
Earlier work this paper cites.
T. D. Kulkarni, A. Saeedi, S. Gautam, and S. J. Gershman, “Deep successor reinforcement learning,”
Original
2016
Earlier work this paper cites.
L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real nvp,”
Original
2016
Earlier work this paper cites.
A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves
2016
Earlier work this paper cites.
M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. Mcgrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight Experience Replay,” in
2017
Earlier work this paper cites.
A. Levy, R. Platt, and K. Saenko, “Hierarchical Actor-Critic,”
Original
2017
Earlier work this paper cites.
P. Rauber, F. Mutz, and J. J. Schmidhuber, “Hindsight policy gradients,” in
2017
Earlier work this paper cites.

Similar

A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver, “Successor features for transfer in reinforcement learning,” in
2017
Cited alongside, same era.
D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell, “Zero-Shot Visual Imitation,” in
2018
Cited alongside, same era.
V. Dhiman, S. Banerjee, J. M. Siskind, and J. J. Corso, “Floyd-warshall reinforcement learning: Learning from past experiences to reach new goals,”
Original
2018
Cited alongside, same era.
V. Pong, S. Gu, M. Dalal, and S. Levine, “Temporal Difference Models: Model-Free Deep RL For Model-Based Control,” in
2018
Cited alongside, same era.
O. Nachum, G. Brain, S. Gu, H. Lee, and S. Levine, “Data-Efficient Hierarchical Reinforcement Learning,” in
2018
Cited alongside, same era.
A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine, “Visual Reinforcement Learning with Imagined Goals,” in
2018
Cited alongside, same era.
C. Florensa, D. Held, X. Geng, and P. Abbeel, “Automatic goal generation for reinforcement learning agents,” in
2018
Cited alongside, same era.
M. Plappert, M. Andrychowicz, A. Ray
Original
2018
Cited alongside, same era.
A. Laversanne-Finot, A. Pere, and P.-Y. Oudeyer, “Curiosity driven exploration of learned disentangled goal spaces,” in
2018
Cited alongside, same era.
F. Torabi, G. Warnell, and P. Stone, “Generative adversarial imitation from observation,”
Original
2018
Cited alongside, same era.
J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine, “Variational inverse control with events: A general framework for data-driven reward definition,” in
2018
Cited alongside, same era.
F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” in
2018
Cited alongside, same era.

Then

A. Barreto, D. Borsa, J. Quan, T. Schaul, D. Silver, M. Hessel, D. Mankowitz, A. Zidek, and R. Munos, “Transfer in deep reinforcement learning using successor features and generalised policy improvement,” in
2018
Later among the works it cites.
D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” in
2018
Later among the works it cites.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” in
2018
Later among the works it cites.
D. Warde-Farley, T. Van De Wiele, T. Kulkarni, C. Ionescu, S. Hansen, and M. Volodymyr, “Unsupervised Control Through Non-Parametric Discriminative Rewards,” in
2019
Later among the works it cites.
C. Colas, P. Fournier, M. Chetouani, O. Sigaud, and P.-Y. Oudeyer, “Curious: intrinsically motivated modular multi-goal reinforcement learning,” in
2019
Later among the works it cites.
A. Edwards, H. Sahni, Y. Schroecker, and C. Isbell, “Imitating latent policies from observation,” in
2019
Later among the works it cites.
L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov, “Efficient exploration via state marginal matching,” in
2019
Later among the works it cites.
Y. Lee, E. S. Hu, Z. Yang, A. Yin, and J. J. Lim, “IKEA furniture assembly environment for long-horizon complex manipulation tasks,”
Original
2019
Later among the works it cites.
A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, “Go-explore: a new approach for hard-exploration problems,”
Original
2019
Later among the works it cites.
Z. Ren, K. Dong, Y. Zhou, Q. Liu, and J. Peng, “Exploration via hindsight goal generation,” in
2019
Later among the works it cites.
V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine, “Skew-fit: State-covering self-supervised reinforcement learning,” in
2020
Later among the works it cites.
T. Jurgenson, E. Groshev, and A. Tamar, “Sub-goal trees–a framework for goal-directed trajectory prediction and optimization,” in
2020
Later among the works it cites.
A. Nair, S. Bahl, A. Khazatsky, V. Pong, G. Berseth, and S. Levine, “Contextual imagined goals for self-supervised robotic learning,” in
2020
Later among the works it cites.
S. Nair, S. Savarese, and C. Finn, “Goal-aware prediction: Learning to model what matters,” in
2020
Later among the works it cites.
B. Eysenbach, X. Geng, S. Levine, and R. Salakhutdinov, “Rewriting history with inverse rl: Hindsight inference for policy improvement,” in
2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…