2020

Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

Xiao, Ted, Jang, Eric, Kalashnikov, Dmitry et al.

Understand

We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action.

Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed.
In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays.
We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms.

Built on

Stochastic processes , volume 2
Sheldon M Ross, John J Kelly, Roger J Sullivan, William James Perry, Donald Mercer, Ruth M Davis, Thomas Dell Washburn, Earl V Sager, Joseph B Boyce, and Vincent L Bristow · 1996
Earlier work this paper cites.
Reinforcement learning for continuous stochastic control problems
Rémi Munos and Paul Bourgine · 1998
Earlier work this paper cites.
Reinforcement Learning: An Introduction
Richard S. Sutton and Andrew G. Barto · 1998
Earlier work this paper cites.
Reinforcement learning in continuous time and space
Kenji Doya · 2000
Earlier work this paper cites.
Reinforcement learning using neural networks, with applications to motor control
Rémi Coulom · 2002
Earlier work this paper cites.
Path integrals and symmetry breaking for optimal control theory
H J Kappen · 2005
Earlier work this paper cites.
An application of reinforcement learning to aerobatic helicopter flight
Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Y. Ng · 2006
Earlier work this paper cites.
Planning and learning in environments with delayed feedback
Thomas J. Walsh, Ali Nouri, Lihong Li, and Michael L. Littman · 2007
Earlier work this paper cites.
Spike-based reinforcement learning in continuous state and action space: When policy gradient methods fail
Eleni Vasilaki, Nicolas Frémaux, Robert Urbanczik, Walter Senn, and Wulfram Gerstner · 2009
Earlier work this paper cites.
Control delay in reinforcement learning for real-time dynamic systems: A memoryless approach
Erik Schuitema, Lucian Busoniu, Robert Babuka, and Pieter P. Jonker · 2010
Earlier work this paper cites.

Similar

Reinforcement learning of motor skills in high dimensions: A path integral approach
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal · 2010
Cited alongside, same era.
Reinforcement learning using a continuous time actor-critic framework with spiking neurons
Nicolas Frémaux, Henning Sprekeler, and Wulfram Gerstner · 2013
Cited alongside, same era.
Playing atari with deep reinforcement learning
Original
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller · 2013
Cited alongside, same era.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis · 2015
Cited alongside, same era.
Openai gym, 2016
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis · 2016
Cited alongside, same era.
Using convolutional neural networks in robots with limited computational resources: Detecting nao robots while playing soccer
Original
Nicolás Cruz, Kenzo Lobos-Tsunekawa, and Javier Ruiz del Solar · 2017
Cited alongside, same era.
Motion perception in reinforcement learning with dynamic objects
Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, and Thomas Brox · 2018
Cited alongside, same era.
At Human Speed: Deep Reinforcement Learning with Action Delay
Vlad Firoiu, Tina Ju, and Joshua Tenenbaum · 2018
Cited alongside, same era.
Tf-agents: A library for reinforcement learning in tensorflow, 2018
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Chris Harris, Vincent Vanhoucke, et al · 2018
Cited alongside, same era.

Then

Soft actor-critic algorithms and applications
Original
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine · 2018
Later among the works it cites.
Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation
Original
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine · 2018
Later among the works it cites.
Learning temporal point processes via reinforcement learning
Shuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, and Le Song · 2018
Later among the works it cites.
Adaptive skip intervals: Temporal abstraction for recurrent dynamical models
Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, and Bernhard Schölkopf · 2018
Later among the works it cites.
Learning dexterous in-hand manipulation
Original
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba · 2018
Later among the works it cites.
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke · 2018
Later among the works it cites.
Deepmind control suite
Original
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller · 2018
Later among the works it cites.
Deep reinforcement learning of marked temporal point processes
Utkarsh Upadhyay, Abir De, and Manuel Gomez-Rodrizuez · 2018
Later among the works it cites.
Real-time reinforcement learning
Simon Ramstedt and Chris Pal · 2019
Later among the works it cites.
Making Deep Q-learning Methods Robust to Time Discretization
Correntin Tallec, Leonard Blier, and Yann Ollivier · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…