2018

Feedback Control For Cassie With Deep Reinforcement Learning

Xie, Zhaoming, Berseth, Glen, Clary, Patrick et al.

Understand

Bipedal locomotion skills are challenging to develop.

Control strategies often use local linearization of the dynamics in conjunction with reduced-order abstractions to yield tractable solutions.
In these model-based control strategies, the controller is often not fully aware of many details, including torque limits, joint limits, and other non-linearities that are necessarily excluded from the control computations for simplicity.
Deep reinforcement learning (DRL) offers a promising model-free approach for controlling bipedal locomotion which can more fully exploit the dynamics.

Built on

R. Tedrake, T. W. Zhang, and H. S. Seung, “Learning to walk in 20 minutes,” in Proceedings of the Fourteenth Yale Workshop on Adaptive and Learning Systems , vol. 95585. Yale University New Haven (CT), 2005, pp. 1939–1412
1939
Earlier work this paper cites.
R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Proceedings of the 12th International Conference on Neural Information Processing Systems , ser. NIPS’99. Cambridge, MA, USA: MIT Press, 1999, pp. 1057–1063. [Online]. Available: http://dl.acm.org/citation.cfm?id=3009657.3009806
1999
Earlier work this paper cites.
E. Schuitema, M. Wisse, T. Ramakers, and P. Jonker, “The design of leo: A 2d bipedal walking robot for online autonomous reinforcement learning,” in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems , Oct 2010, pp. 3238–3243
2010
Earlier work this paper cites.
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , Oct 2012, pp. 5026–5033
2012
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR , vol. abs/1412.6980, 2014. [Online]. Available: http://arxiv.org/abs/1412.6980
Original
2014
Earlier work this paper cites.
S. C. Hsu, X. Xu, and A. D. Ames, “Control barrier function based quadratic programs with application to bipedal robotic walking,” in 2015 American Control Conference (ACC) , July 2015, pp. 4542–4548
2015
Earlier work this paper cites.
M. Posa, S. Kuindersma, and R. Tedrake, “Optimization and stabilization of trajectories for constrained dynamical systems,” in 2016 IEEE International Conference on Robotics and Automation (ICRA) , May 2016, pp. 1366–1373
2016
Earlier work this paper cites.

Similar

X. Da, O. Harib, R. Hartley, B. Griffin, and J. W. Grizzle, “From 2d design of underactuated bipedal gaits to 3d implementation: Walking with speed tracking,” IEEE Access , vol. 4, pp. 3469–3478, 2016
2016
Cited alongside, same era.
A. Hereid, E. A. Cousineau, C. M. Hubicki, and A. D. Ames, “3d dynamic walking with underactuated humanoid robots: A direct collocation framework for optimizing hybrid zero dynamics,” in 2016 IEEE International Conference on Robotics and Automation (ICRA) , May 2016, pp. 1447–1454
2016
Cited alongside, same era.
X. B. Peng, G. Berseth, K. Yin, and M. Van De Panne, “Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning,” ACM Trans. Graph. , vol. 36, no. 4, pp. 41:1–41:13, July 2017. [Online]. Available: http://doi.acm.org/10.1145/3072959.3073602
2017
Cited alongside, same era.
X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” CoRR , vol. abs/1710.06537, 2017. [Online]. Available: http://arxiv.org/abs/1710.06537
Original
2017
Cited alongside, same era.
N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A. Eslami, M. A. Riedmiller, and D. Silver, “Emergence of locomotion behaviours in rich environments,” CoRR , vol. abs/1707.02286, 2017. [Online]. Available: http://arxiv.org/abs/1707.02286
Original
2017
Cited alongside, same era.
J. Merel, Y. Tassa, D. TB, S. Srinivasan, J. Lemmon, Z. Wang, G. Wayne, and N. Heess, “Learning human behaviors from motion capture by adversarial imitation,” CoRR , vol. abs/1707.02201, 2017. [Online]. Available: http://arxiv.org/abs/1707.02201
Original
2017
Cited alongside, same era.
Z. Wang, J. Merel, S. E. Reed, G. Wayne, N. de Freitas, and N. Heess, “Robust imitation of diverse behaviors,” CoRR , vol. abs/1707.02747, 2017. [Online]. Available: http://arxiv.org/abs/1707.02747
Original
2017
Cited alongside, same era.
S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” The International Journal of Robotics Research , vol. 0, no. 0, p. 0278364917710318, 0. [Online]. Available: https://doi.org/10.1177/0278364917710318
Cited in the paper.

Then

X. B. Peng and M. van de Panne, “Learning locomotion skills using deeprl: Does the choice of action space matter?” in Proc. ACM SIGGRAPH / Eurographics Symposium on Computer Animation , 2017
2017
Later among the works it cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” CoRR , vol. abs/1707.06347, 2017. [Online]. Available: http://arxiv.org/abs/1707.06347
Original
2017
Later among the works it cites.
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” CoRR , vol. abs/1709.06560, 2017. [Online]. Available: http://arxiv.org/abs/1709.06560
Original
2017
Later among the works it cites.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017
2017
Later among the works it cites.
W. Yu, G. Turk, and C. K. Liu, “Learning Symmetry and Low-energy Locomotion,” ArXiv e-prints: arXiv:1801.08093v1 , Jan. 2018
Original
2018
Closest in time.
R. Tedrake, “Underactuated robotics: Algorithms for walking, running, swimming, flying, and manipulation (course notes for mit 6.832).” 2018
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…