2020

Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning

Seo, Younggyo, Lee, Kimin, Clavera, Ignasi et al.

Understand

Model-based reinforcement learning (RL) has shown great potential in various control tasks in terms of both sample-efficiency and final performance.

However, learning a generalizable dynamics model robust to changes in dynamics remains a challenge since the target transition dynamics follow a multi-modal distribution.
In this paper, we present a new model-based RL algorithm, coined trajectory-wise multiple choice learning, that learns a multi-headed dynamics model for dynamics generalization.
The main idea is updating the most accurate prediction head to specialize each head in certain environments with similar dynamics, i.e., clustering environments.

Built on

Integrated architectures for learning, planning, and reacting based on approximating dynamic programming
Sutton, Richard S · 1990
Earlier work this paper cites.
Improving transient response of adaptive control systems using multiple models and switching
Narendra, Kumpati S and Balakrishnan, Jeyendran · 1994
Earlier work this paper cites.
A comparison of direct and model-based reinforcement learning
Atkeson, Christopher G and Santamaria, Juan Carlos · 1997
Earlier work this paper cites.
Predictive control: with constraints
Maciejowski, Jan Marian · 2002
Earlier work this paper cites.
A tutorial on the cross-entropy method
De Boer, Pieter-Tjerk, Kroese, Dirk P, Mannor, Shie, and Rubinstein, Reuven Y · 2005
Earlier work this paper cites.
Visualizing data using t-sne
Maaten, Laurens van der and Hinton, Geoffrey · 2008
Earlier work this paper cites.
Multiple choice learning: Learning to produce multiple structured outputs
Guzman-Rivera, Abner, Batra, Dhruv, and Kohli, Pushmeet · 2012
Earlier work this paper cites.
Synthesis and stabilization of complex behaviors through online trajectory optimization
Tassa, Yuval, Erez, Tom, and Todorov, Emanuel · 2012
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Todorov, Emanuel, Erez, Tom, and Tassa, Yuval · 2012
Earlier work this paper cites.
A survey on policy search for robotics
Deisenroth, Marc Peter, Neumann, Gerhard, and Peters, Jan · 2013
Earlier work this paper cites.
Efficiently enforcing diversity in multi-output structured prediction
Guzman-Rivera, Abner, Kohli, Pushmeet, Batra, Dhruv, and Rutenbar, Rob · 2014
Earlier work this paper cites.
Learning neural network policies with guided policy search under unknown dynamics
Levine, Sergey and Abbeel, Pieter · 2014
Earlier work this paper cites.
Predicting multiple structured visual interpretations
Dey, Debadeepta, Ramakrishna, Varun, Hebert, Martial, and Andrew Bagnell, J · 2015
Earlier work this paper cites.
Deepmpc: Learning deep latent features for model predictive control
Lenz, Ian, Knepper, Ross A, and Saxena, Ashutosh · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, et al · 2015
Earlier work this paper cites.

Similar

Trust region policy optimization
Schulman, John, Levine, Sergey, Abbeel, Pieter, Jordan, Michael, and Moritz, Philipp · 2015
Cited alongside, same era.
Openai gym
Original
Brockman, Greg, Cheung, Vicki, Pettersson, Ludwig, Schneider, Jonas, Schulman, John, Tang, Jie, and Zaremba, Wojciech · 2016
Cited alongside, same era.
Stochastic multiple choice learning for training diverse deep ensembles
Lee, Stefan, Prakash, Senthil Purushwalkam Shiva, Cogswell, Michael, Ranjan, Viresh, Crandall, David, and Batra, Dhruv · 2016
Cited alongside, same era.
End-to-end training of deep visuomotor policies
Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter · 2016
Cited alongside, same era.
Continuous control with deep reinforcement learning
Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan · 2016
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Finn, Chelsea, Abbeel, Pieter, and Levine, Sergey · 2017
Cited alongside, same era.
Confident multiple choice learning
Lee, Kimin, Hwang, Changho, Park, Kyoung Soo, and Shin, Jinwoo · 2017
Cited alongside, same era.
Mastering the game of go without human knowledge
Silver, David, Schrittwieser, Julian, Simonyan, Karen, Antonoglou, Ioannis, Huang, Aja, Guez, Arthur, Hubert, Thomas, Baker, Lucas, Lai, Matthew, Bolton, Adrian, et al · 2017
Cited alongside, same era.
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, Kurtland, Calandra, Roberto, McAllister, Rowan, and Levine, Sergey · 2018
Cited alongside, same era.
Model-based reinforcement learning via meta-policy optimization
Clavera, Ignasi, Rothfuss, Jonas, Schulman, John, Fujita, Yasuhiro, Asfour, Tamim, and Abbeel, Pieter · 2018
Cited alongside, same era.
Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation
Kalashnikov, Dmitry, Irpan, Alex, Pastor, Peter, Ibarz, Julian, Herzog, Alexander, Jang, Eric, Quillen, Deirdre, Holly, Ethan, Kalakrishnan, Mrinal, Vanhoucke, Vincent, et al · 2018
Cited alongside, same era.
Model-ensemble trust-region policy optimization
Kurutach, Thanard, Clavera, Ignasi, Duan, Yan, Tamar, Aviv, and Abbeel, Pieter · 2018
Cited alongside, same era.

Then

Graph networks as learnable physics engines for inference and control
Sanchez-Gonzalez, Alvaro, Heess, Nicolas, Springenberg, Jost Tobias, Merel, Josh, Riedmiller, Martin, Hadsell, Raia, and Battaglia, Peter · 2018
Later among the works it cites.
Reinforcement learning: An introduction
Sutton, Richard S and Barto, Andrew G · 2018
Later among the works it cites.
Task-agnostic dynamics priors for deep reinforcement learning
Du, Yilun and Narasimhan, Karthik · 2019
Later among the works it cites.
Learning latent dynamics for planning from pixels
Hafner, Danijar, on Robotics, LillInternational Conference, Automationp, Timothy, Fischer, Ian, Villegas, Ruben, Ha, David, Lee, Honglak, and Davidson, James · 2019
Later among the works it cites.
When to trust your model: Model-based policy optimization
Janner, Michael, Fu, Justin, Zhang, Marvin, and Levine, Sergey · 2019
Later among the works it cites.
Efficient off-policy meta-reinforcement learning via probabilistic context variables
Rakelly, Kate, Zhou, Aurick, Quillen, Deirdre, Finn, Chelsea, and Levine, Sergey · 2019
Later among the works it cites.
Mastering atari, go, chess and shogi by planning with a learned model
Original
Schrittwieser, Julian, Antonoglou, Ioannis, Hubert, Thomas, Simonyan, Karen, Sifre, Laurent, Schmitt, Simon, Guez, Arthur, Lockhart, Edward, Hassabis, Demis, Graepel, Thore, et al · 2019
Later among the works it cites.
Versatile multiple choice learning and its application to vision computing
Tian, Kai, Xu, Yi, Zhou, Shuigeng, and Guan, Jihong · 2019
Later among the works it cites.
Grandmaster level in starcraft ii using multi-agent reinforcement learning
Vinyals, Oriol, Babuschkin, Igor, Czarnecki, Wojciech M, Mathieu, Michael, Dudzik, Andrew, Chung, Junyoung, Choi, David H, Powell, Richard, Ewalds, Timo, Georgiev, Petko, et al · 2019
Later among the works it cites.
Environment probing interaction policies
Zhou, Wenxuan, Pinto, Lerrel, and Gupta, Abhinav · 2019
Later among the works it cites.
Model-based meta-reinforcement learning for flight with suspended payloads
Original
Belkhale, Suneel, Li, Rachel, Kahn, Gregory, McAllister, Rowan, Calandra, Roberto, and Levine, Sergey · 2020
Closest in time.
Dream to control: Learning behaviors by latent imagination
Hafner, Danijar, on Robotics, LillInternational Conference, Automationp, Timothy, Ba, Jimmy, and Norouzi, Mohammad · 2020
Closest in time.
Context-aware dynamics model for generalization in model-based reinforcement learning
Lee, Kimin, Seo, Younggyo, Lee, Seunghyun, Lee, Honglak, and Shin, Jinwoo · 2020
Closest in time.
Exploring model-based planning with policy networks
Wang, Tingwu and Ba, Jimmy · 2020
Closest in time.
Dynamics-aware embeddings
Whitney, William, Agarwal, Rajat, Cho, Kyunghyun, and Gupta, Abhinav · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…