2018

Residual Reinforcement Learning for Robot Control

Johannink, Tobias, Bahl, Shikhar, Nair, Ashvin et al.

Understand

Conventional feedback control methods can solve various types of robot control problems very efficiently by capturing the structure with explicit models, such as rigid body equations of motion.

However, many control problems in modern manufacturing deal with contacts and friction, which are difficult to capture with first-order physical modeling.
Hence, applying control design methodologies to these kinds of problems often results in brittle and inaccurate controllers, which have to be manually tuned for deployment.
Reinforcement learning (RL) methods have been demonstrated to be capable of learning continuous robot controllers from interactions with the environment, even for problems that include friction and contacts.

Built on

A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in
1999
Earlier work this paper cites.
J. Frank, S. Mannor, and D. Precup, “Reinforcement Learning in the Presence of Rare Events,” in
2008
Earlier work this paper cites.
J. Peters and S. Schaal, “Reinforcement learning of motor skills with policy gradients,”
2008
Earlier work this paper cites.
J. Kober and J. Peter, “Policy search for motor primitives in robotics,” in
2008
Earlier work this paper cites.
J. Peters, K. Mülling, and Y. Altün, “Relative Entropy Policy Search,” in
2010
Earlier work this paper cites.
M. P. Deisenroth and C. E. Rasmussen, “PILCO: A model-based and data-efficient approach to policy search,” in
2011
Earlier work this paper cites.
E. Todorov, T. Erez, and Y. Tassa, “MuJoCo: A physics engine for model-based control,” in
2012
Earlier work this paper cites.
L. Torrey and M. E. Taylor, “Teaching on a Budget: Agents Advising Agents in Reinforcement Learning,” in
2013
Earlier work this paper cites.
C. Daniel, M. Viering, J. Metz, O. Kroemer, and J. Peters, “Active Reward Learning,” in
2014
Earlier work this paper cites.
R. Loftin, J. Macglashan, B. Peng, M. E. Taylor, M. L. Littman, J. Huang, and D. L. Roberts, “A Strategy-Aware Technique for Learning Behaviors from Discrete Human Feedback,” in
2014
Earlier work this paper cites.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in
2016
Earlier work this paper cites.

Similar

S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-End Training of Deep Visuomotor Policies,”
2016
Cited alongside, same era.
K. Subramanian, C. L. Isbell, and A. L. Thomaz, “Exploration from Demonstration for Interactive Reinforcement Learning,” in
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in
2016
Cited alongside, same era.
U. Kannengiesser, R. Heininger, L. Billy, P. Terpak, M. Neubauer, C. Stary, D. Majoe, A. Totter, and D. Bonaldi,
2017
Cited alongside, same era.
L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta, “Robust Adversarial Reinforcement Learning,” in
2017
Cited alongside, same era.
S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection,”
2017
Cited alongside, same era.
S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates,”
2017
Cited alongside, same era.
I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-maron, M. Vecerik, T. Lampe, Y. Tassa, T. Erez, M. Riedmiller, and M. R. Deepmind, “Data-efficient Deep Reinforcement Learning for Dexterous Manipulation,”
2017
Cited alongside, same era.
P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in
2017
Cited alongside, same era.
M. Večerík, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller, “Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards,”
2017
Cited alongside, same era.
S. Fujimoto, H. van Hoof, and D. Meger, “Addressing Function Approximation Error in Actor-Critic Methods,”
Original
2018
Cited alongside, same era.

Then

V. Pong, S. Gu, M. Dalal, and S. Levine, “Temporal Difference Models: Model-Free Deep RL For Model-Based Control,” in
2018
Closest in time.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” in
2018
Closest in time.
G. Thomas, M. Chien, A. Tamar, J. A. Ojea, and P. Abbeel, “Learning Robotic Assembly from CAD,” in
2018
Closest in time.
W. Saunders, G. Sastry, A. Stuhlmüller, and O. Evans, “Trial without Error: Towards Safe Reinforcement Learning via Human Intervention,” in
2018
Closest in time.
A. Rajeswaran, V. Kumar, A. Gupta, J. Schulman, E. Todorov, and S. Levine, “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations,” in
2018
Closest in time.
T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, J. Z. Leibo, and A. Gruslys, “Learning from Demonstrations for Real World Reinforcement Learning,” in
2018
Closest in time.
A. Nair, B. Mcgrew, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Overcoming Exploration in Reinforcement Learning with Demonstrations,” in
2018
Closest in time.
M. Srouji, J. Zhang, and R. Salakhutdinov, “Structured Control Nets for Deep Reinforcement Learning,” in
2018
Closest in time.
T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling, “Residual Policy Learning,”
Original
2018
Closest in time.
F. Wang and J. Decker, “Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator,”
Original
2018
Closest in time.
N. Sünderhauf, O. Brock, W. Scheirer, R. Hadsell, D. Fox, J. Leitner, B. Upcroft, P. Abbeel, W. Burgard, M. Milford, and P. Corke, “The Limits and Potentials of Deep Learning for Robotics,”
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…