2022

Safe Reinforcement Learning for Legged Locomotion

Yang, Tsung-Yen, Zhang, Tingnan, Luu, Linda et al.

Understand

Designing control policies for legged locomotion is complex due to the under-actuated and non-continuous robot dynamics.

Model-free reinforcement learning provides promising tools to tackle this challenge.
However, a major bottleneck of applying model-free reinforcement learning in real world is safety.
In this paper, we propose a safe reinforcement learning framework that switches between a safe recovery policy that prevents the robot from entering unsafe states, and a learner policy that is optimized to complete the task.

Built on

E. Altman, Constrained Markov decision processes . CRC Press, 1999
1999
Earlier work this paper cites.
N. Kohl and P. Stone, “Policy gradient reinforcement learning for fast quadrupedal locomotion,” in IEEE International Conference on Robotics and Automation, 2004. Proceedings , 2004
2004
Earlier work this paper cites.
R. Tedrake, T. W. Zhang, and H. S. Seung, “Stochastic policy gradient reinforcement learning on a simple 3d biped,” in IEEE/RSJ International Conference on Intelligent Robots and Systems , 2004
2004
Earlier work this paper cites.
G. Endo, J. Morimoto, T. Matsubara, J. Nakanishi, and G. Cheng, “Learning cpg sensory feedback with policy gradient for biped locomotion for a full-body humanoid.” 2005
2005
Earlier work this paper cites.
J. Garcia and F. Fernandez, “A comprehensive survey on safe reinforcement learning,” Journal of Machine Learning Research , 2015
2015
Earlier work this paper cites.
J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz, “Trust region policy optimization,” in Proceedings of the 32nd International Conference on Machine Learning, ICML 2015 . JMLR.org, 2015
2015
Earlier work this paper cites.
S. Junges, N. Jansen, C. Dehnert, U. Topcu, and J.-P. Katoen, “Safety-constrained reinforcement learning for mdps,” in International Conference on Tools and Algorithms for the Construction and Analysis of Systems . Springer, 2016
2016
Earlier work this paper cites.
J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in Proceedings of the 34th International Conference on Machine Learning, ICML , 2017
2017
Earlier work this paper cites.
F. Berkenkamp, M. Turchetta, A. P. Schoellig, and A. Krause, “Safe model-based reinforcement learning with stability guarantees,” in Advances in Neural Information Processing Systems 30 , 2017
2017
Earlier work this paper cites.
E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation in robotics, games and machine learning,” 2017
2017
Earlier work this paper cites.
T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine, “Learning to walk via deep reinforcement learning,” arXiv preprint arXiv:1812.11103 , 2018
Original
2018
Earlier work this paper cites.
Y. Chow, O. Nachum, E. A. Duéñez-Guzmán, and M. Ghavamzadeh, “A lyapunov-based approach to safe reinforcement learning,” in Advances in Neural Information Processing Systems 31, NeurIPS , 2018
2018
Earlier work this paper cites.
G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, and Y. Tassa, “Safe exploration in continuous action spaces,” arXiv preprint arXiv:1801.08757 , 2018
Original
2018
Earlier work this paper cites.
M. Fu and A. Prashanth L, “Risk-sensitive reinforcement learning: A constrained optimization viewpoint,” arXiv preprint arXiv:1810.09126 , 2018
Original
2018
Earlier work this paper cites.
J. F. Fisac, A. K. Akametalu, M. N. Zeilinger, S. Kaynama, J. Gillula, and C. J. Tomlin, “A general safety framework for learning-based control in uncertain robotic systems,” IEEE Transactions on Automatic Control , no. 7, 2018
2018
Earlier work this paper cites.

Similar

T. Koller, F. Berkenkamp, M. Turchetta, and A. Krause, “Learning-based model predictive control for safe exploration,” in 2018 IEEE conference on decision and control (CDC) . IEEE, 2018
2018
Cited alongside, same era.
L. Wang, E. A. Theodorou, and M. Egerstedt, “Safe learning of quadrotor dynamics using barrier certificates,” in IEEE International Conference on Robotics and Automation (ICRA) , 2018
2018
Cited alongside, same era.
J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim, “Dynamic locomotion in the mit cheetah 3 through convex model-predictive control,” in 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) . IEEE, 2018
2018
Cited alongside, same era.
I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas et al. , “Solving rubik’s cube with a robot hand,” arXiv preprint arXiv:1910.07113 , 2019
Original
2019
Cited alongside, same era.
C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,” in 7th International Conference on Learning Representations, ICLR 2019 . OpenReview.net, 2019
2019
Cited alongside, same era.
Y. Chow, O. Nachum, A. Faust, M. Ghavamzadeh, and E. Duenez-Guzman, “Lyapunov-based safe policy optimization for continuous control,” arXiv preprint arXiv:1901.10031 , 2019
Original
2019
Cited alongside, same era.
S. Fujimoto, E. Conti, M. Ghavamzadeh, and J. Pineau, “Benchmarking batch deep reinforcement learning algorithms,” arXiv preprint arXiv:1910.01708 , 2019
Original
2019
Cited alongside, same era.
G. Avni, R. Bloem, K. Chatterjee, T. A. Henzinger, B. Könighofer, and S. Pranger, “Run-time optimization for learned controllers through quantitative games,” in International Conference on Computer Aided Verification . Springer, 2019
2019
Cited alongside, same era.
G. Shi, X. Shi, M. O’Connell, R. Yu, K. Azizzadenesheli, A. Anandkumar, Y. Yue, and S.-J. Chung, “Neural lander: Stable drone landing control using learned dynamics,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019
2019
Cited alongside, same era.
W. Yu, V. C. Kumar, G. Turk, and C. K. Liu, “Sim-to-real transfer for biped locomotion,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2019
2019
Cited alongside, same era.
J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” Science robotics , no. 47, 2020
2020
Cited alongside, same era.
S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan, “Learning to walk in the real world with minimal human effort,” arXiv preprint arXiv:2002.08550 , 2020
Original
2020
Cited alongside, same era.

Then

M. Hasanbeig, A. Abate, and D. Kroening, “Cautious reinforcement learning with logical constraints,” arXiv preprint arXiv:2002.12156 , 2020
Original
2020
Later among the works it cites.
N. Jansen, B. Könighofer, S. Junges, A. Serban, and R. Bloem, “Safe Reinforcement Learning Using Probabilistic Shields (Invited Paper),” in 31st International Conference on Concurrency Theory , 2020
2020
Later among the works it cites.
K. Srinivasan, B. Eysenbach, S. Ha, J. Tan, and C. Finn, “Learning to be safe: Deep rl with a safety critic,” arXiv preprint arXiv:2010.14603 , 2020
Original
2020
Later among the works it cites.
T. Yang, J. Rosca, K. Narasimhan, and P. J. Ramadge, “Projection-based constrained policy optimization,” in 8th International Conference on Learning Representations, ICLR . OpenReview.net, 2020
2020
Later among the works it cites.
A. Stooke, J. Achiam, and P. Abbeel, “Responsive safety in reinforcement learning by PID lagrangian methods,” in Proceedings of the 37th International Conference on Machine Learning, ICML , 2020
2020
Later among the works it cites.
L. Zheng and L. J. Ratliff, “Constrained upper confidence reinforcement learning,” arXiv preprint arXiv:2001.09377 , 2020
Original
2020
Later among the works it cites.
L. Hewing, K. P. Wabersich, M. Menner, and M. N. Zeilinger, “Learning-based model predictive control: Toward safe learning in control,” Annual Review of Control, Robotics, and Autonomous Systems , 2020
2020
Later among the works it cites.
X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” arXiv preprint arXiv:2004.00784 , 2020
Original
2020
Later among the works it cites.
W. Yu, J. Tan, Y. Bai, E. Coumans, and S. Ha, “Learning fast adaptation with meta strategy optimization,” IEEE Robotics and Automation Letters , no. 2, 2020
2020
Later among the works it cites.
B. Thananjeyan, A. Balakrishna, S. Nair, M. Luo, K. Srinivasan, M. Hwang, J. E. Gonzalez, J. Ibarz, C. Finn, and K. Goldberg, “Recovery rl: Safe reinforcement learning with learned recovery zones,” IEEE Robotics and Automation Letters , no. 3, 2021
2021
Later among the works it cites.
“Unitree laikago,” https://www.unitree.com/ , accessed: 2021
2021
Later among the works it cites.
“Unitree a1,” https://www.unitree.com/products/a1/ , accessed: 2021
2021
Later among the works it cites.
O. Bastani, S. Li, and A. Xu, “Safe reinforcement learning via statistical model predictive shielding,” in Robotics: Science and Systems , 2021
2021
Later among the works it cites.
H. Chen, Z. Hong, S. Yang, P. M. Wensing, and W. Zhang, “Quadruped capturability and push recovery via a switched-systems characterization of dynamic balance,” arXiv preprint arXiv:2201.11928 , 2022
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…