Built on

M. H. Raibert, Legged robots that balance . MIT press, 1986
1986
Earlier work this paper cites.
J. Z. Kolter, P. Abbeel, and A. Y. Ng, “Hierarchical apprenticeship learning with application to quadruped locomotion,” in Advances in Neural Information Processing Systems . Citeseer, 2008, pp. 769–776
2008
Earlier work this paper cites.
M. Raibert, K. Blankespoor, G. Nelson, and R. Playter, “Bigdog, the rough-terrain quadruped robot,” IFAC Proceedings Volumes , vol. 41, no. 2, pp. 10 822–10 825, 2008
2008
Earlier work this paper cites.
K. E. Adolph, W. G. Cole, M. Komati, J. S. Garciaguirre, D. Badaly, J. M. Lingeman, G. L. Chan, and R. B. Sotsky, “How do you learn to walk? thousands of steps and dozens of falls per day,” Psychological science , vol. 23, no. 11, pp. 1387–1394, 2012
2012
Earlier work this paper cites.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015
Original
2015
Earlier work this paper cites.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438 , 2015
Original
2015
Earlier work this paper cites.
R. Antonova, S. Cruciani, C. Smith, and D. Kragic, “Reinforcement learning for pivoting task,” arXiv preprint arXiv:1703.00472 , 2017
Original
2017
Earlier work this paper cites.
X. B. Peng and M. van de Panne, “Learning locomotion skills using deeprl: Does the choice of action space matter?” in Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation , 2017, pp. 1–13
2017
Earlier work this paper cites.
L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta, “Robust adversarial reinforcement learning,” in International Conference on Machine Learning . PMLR, 2017, pp. 2817–2826
2017
Earlier work this paper cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017
Original
2017
Earlier work this paper cites.
J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim, “Dynamic locomotion in the mit cheetah 3 through convex model-predictive control,” in 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) . IEEE, 2018, pp. 1–9
2018
Earlier work this paper cites.
X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in 2018 IEEE international conference on robotics and automation (ICRA) . IEEE, 2018, pp. 3803–3810
2018
Earlier work this paper cites.
J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, “Sim-to-real: Learning agile locomotion for quadruped robots,” in Proceedings of Robotics: Science and Systems , Pittsburgh, Pennsylvania, June 2018
2018
Earlier work this paper cites.
J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Science Robotics , vol. 4, no. 26, 2019
2019
Earlier work this paper cites.

Similar

J. Lee, J. Hwangbo, and M. Hutter, “Robust recovery controller for a quadrupedal robot using deep reinforcement learning,” arXiv preprint arXiv:1901.07517 , 2019
Original
2019
Cited alongside, same era.
P. Varin, L. Grossman, and S. Kuindersma, “A comparison of action spaces for learning manipulation tasks,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2019, pp. 6015–6021
2019
Cited alongside, same era.
G. Bellegarda and Q. Nguyen, “Robust quadruped jumping via deep reinforcement learning,” arXiv preprint arXiv:2011.07089 , 2020
Original
2020
Cited alongside, same era.
S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis, “Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control,” arXiv preprint arXiv:2012.03094 , 2020
Original
2020
Cited alongside, same era.

Then

V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. , “Isaac gym: High performance gpu-based physics simulation for robot learning,” arXiv preprint arXiv:2108.10470 , 2021
Original
2021
Later among the works it cites.
J. Siekmann, Y. Godse, A. Fern, and J. Hurst, “Sim-to-real learning of all common bipedal gaits via periodic reward composition,” in 2021 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 7309–7315
2021
Later among the works it cites.
L. Smith, J. C. Kew, X. B. Peng, S. Ha, J. Tan, and S. Levine, “Legged robots that keep on learning: Fine-tuning locomotion policies in the real world,” arXiv preprint arXiv:2110.05457 , 2021
Original
2021
Later among the works it cites.
Y. Sun, W. L. Ubellacker, W.-L. Ma, X. Zhang, C. Wang, N. V. Csomay-Shanklin, M. Tomizuka, K. Sreenath, and A. D. Ames, “Online learning of unknown dynamics for model-based controllers in legged locomotion,” IEEE Robotics and Automation Letters , vol. 6, no. 4, pp. 8442–8449, 2021

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2022

Learning Torque Control for Quadrupedal Locomotion

Chen, Shuxiao, Zhang, Bike, Mueller, Mark W. et al.

Understand

Reinforcement learning (RL) has become a promising approach to developing controllers for quadrupedal robots.

Conventionally, an RL design for locomotion follows a position-based paradigm, wherein an RL policy outputs target joint positions at a low frequency that are then tracked by a high-frequency proportional-derivative (PD) controller to produce joint torques.
In contrast, for the model-based control of quadrupedal locomotion, there has been a paradigm shift from position-based control to torque-based control.
In light of the recent advances in model-based control, we explore an alternative to the position-based RL paradigm, by introducing a torque-based RL framework, where an RL policy directly predicts joint torques at a high frequency, thus circumventing the use of a PD controller.

Built on

M. H. Raibert, Legged robots that balance . MIT press, 1986
1986
Earlier work this paper cites.
J. Z. Kolter, P. Abbeel, and A. Y. Ng, “Hierarchical apprenticeship learning with application to quadruped locomotion,” in Advances in Neural Information Processing Systems . Citeseer, 2008, pp. 769–776
2008
Earlier work this paper cites.
M. Raibert, K. Blankespoor, G. Nelson, and R. Playter, “Bigdog, the rough-terrain quadruped robot,” IFAC Proceedings Volumes , vol. 41, no. 2, pp. 10 822–10 825, 2008
2008
Earlier work this paper cites.
K. E. Adolph, W. G. Cole, M. Komati, J. S. Garciaguirre, D. Badaly, J. M. Lingeman, G. L. Chan, and R. B. Sotsky, “How do you learn to walk? thousands of steps and dozens of falls per day,” Psychological science , vol. 23, no. 11, pp. 1387–1394, 2012
2012
Earlier work this paper cites.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015
Original
2015
Earlier work this paper cites.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438 , 2015
Original
2015
Earlier work this paper cites.
R. Antonova, S. Cruciani, C. Smith, and D. Kragic, “Reinforcement learning for pivoting task,” arXiv preprint arXiv:1703.00472 , 2017
Original
2017
Earlier work this paper cites.
X. B. Peng and M. van de Panne, “Learning locomotion skills using deeprl: Does the choice of action space matter?” in Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation , 2017, pp. 1–13
2017
Earlier work this paper cites.
L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta, “Robust adversarial reinforcement learning,” in International Conference on Machine Learning . PMLR, 2017, pp. 2817–2826
2017
Earlier work this paper cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017
Original
2017
Earlier work this paper cites.
J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim, “Dynamic locomotion in the mit cheetah 3 through convex model-predictive control,” in 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) . IEEE, 2018, pp. 1–9
2018
Earlier work this paper cites.
X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in 2018 IEEE international conference on robotics and automation (ICRA) . IEEE, 2018, pp. 3803–3810
2018
Earlier work this paper cites.
J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, “Sim-to-real: Learning agile locomotion for quadruped robots,” in Proceedings of Robotics: Science and Systems , Pittsburgh, Pennsylvania, June 2018
2018
Earlier work this paper cites.
J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Science Robotics , vol. 4, no. 26, 2019
2019
Earlier work this paper cites.

Similar

J. Lee, J. Hwangbo, and M. Hutter, “Robust recovery controller for a quadrupedal robot using deep reinforcement learning,” arXiv preprint arXiv:1901.07517 , 2019
Original
2019
Cited alongside, same era.
P. Varin, L. Grossman, and S. Kuindersma, “A comparison of action spaces for learning manipulation tasks,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2019, pp. 6015–6021
2019
Cited alongside, same era.
G. Bellegarda and Q. Nguyen, “Robust quadruped jumping via deep reinforcement learning,” arXiv preprint arXiv:2011.07089 , 2020
Original
2020
Cited alongside, same era.
S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis, “Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control,” arXiv preprint arXiv:2012.03094 , 2020
Original
2020
Cited alongside, same era.

Then

V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. , “Isaac gym: High performance gpu-based physics simulation for robot learning,” arXiv preprint arXiv:2108.10470 , 2021
Original
2021
Later among the works it cites.
J. Siekmann, Y. Godse, A. Fern, and J. Hurst, “Sim-to-real learning of all common bipedal gaits via periodic reward composition,” in 2021 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 7309–7315
2021
Later among the works it cites.
L. Smith, J. C. Kew, X. B. Peng, S. Ha, J. Tan, and S. Levine, “Legged robots that keep on learning: Fine-tuning locomotion policies in the real world,” arXiv preprint arXiv:2110.05457 , 2021
Original
2021
Later among the works it cites.
Y. Sun, W. L. Ubellacker, W.-L. Ma, X. Zhang, C. Wang, N. V. Csomay-Shanklin, M. Tomizuka, K. Sreenath, and A. D. Ames, “Online learning of unknown dynamics for model-based controllers in legged locomotion,” IEEE Robotics and Automation Letters , vol. 6, no. 4, pp. 8442–8449, 2021

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

X. Jiang, H. Wang, Y. Chen, Z. Wu, L. Wang, B. Zou, Y. Yang, Z. Cui, Y. Cai, T. Yu, C. Lv, and Z. Wu, “Mnn: A universal and efficient inference engine,” in MLSys , 2020

2020

Cited alongside, same era.

J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” Science robotics , vol. 5, no. 47, 2020

2020

Cited alongside, same era.

T. Li, N. Lambert, R. Calandra, F. Meier, and A. Rai, “Learning generalizable locomotion skills with hierarchical reinforcement learning,” in 2020 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2020, pp. 413–419

2020

Cited alongside, same era.

X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine, “Learning Agile Robotic Locomotion Skills by Imitating Animals,” in Proceedings of Robotics: Science and Systems , Corvalis, Oregon, USA, July 2020

2020

Cited alongside, same era.

D. Reda, T. Tao, and M. van de Panne, “Learning to locomote: Understanding how environment design matters for deep reinforcement learning,” in Motion, Interaction and Games , 2020, pp. 1–10

2020

Cited alongside, same era.

——, “Robust high-speed running for quadruped robots via deep reinforcement learning,” arXiv preprint arXiv:2103.06484 , 2021

Original

2021

Cited alongside, same era.

X. Da, Z. Xie, D. Hoeller, B. Boots, A. Anandkumar, Y. Zhu, B. Babich, and A. Garg, “Learning a contact-adaptive controller for robust, efficient legged locomotion,” in Conference on Robot Learning . PMLR, 2021, pp. 883–894

2021

Cited alongside, same era.

D. Han and K. E. Adolph, “The impact of errors in infant development: Falling like a baby,” Developmental science , vol. 24, no. 5, 2021

2021

Cited alongside, same era.

2021

Later among the works it cites.

Z. Xie, X. Da, M. van de Panne, B. Babich, and A. Garg, “Dynamics randomization revisited: A case study for quadrupedal locomotion,” in 2021 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 4955–4961

2021

Later among the works it cites.

Y. Yang, T. Zhang, E. Coumans, J. Tan, and B. Boots, “Fast and efficient locomotion via learned gait transitions,” arXiv preprint arXiv:2104.04644 , 2021

Original

2021

Later among the works it cites.

T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics , vol. 7, no. 62, 2022

2022

Closest in time.

N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Conference on Robot Learning . PMLR, 2022, pp. 91–100

2022

Closest in time.

H. Shi, B. Zhou, H. Zeng, F. Wang, Y. Dong, J. Li, K. Wang, H. Tian, and M. Q.-H. Meng, “Reinforcement learning with evolutionary trajectory generator: A general approach for quadrupedal locomotion,” IEEE Robotics and Automation Letters , 2022

2022

Closest in time.

L. Smith, I. Kostrikov, and S. Levine, “A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning,” arXiv preprint arXiv:2208.07860 , 2022

Original

2022

Closest in time.

P. Wu, A. Escontrela, D. Hafner, K. Goldberg, and P. Abbeel, “Daydreamer: World models for physical robot learning,” arXiv preprint arXiv:2206.14176 , 2022

Original

2022

Closest in time.

D. Zhang, A. Loquercio, X. Wu, A. Kumar, J. Malik, and M. W. Mueller, “A zero-shot adaptive quadcopter controller,” arXiv preprint arXiv:2209.09232 , 2022

Original

2022

Closest in time.

S. Choi, G. Ji, J. Park, H. Kim, J. Mun, J. H. Lee, and J. Hwangbo, “Learning quadrupedal locomotion on deformable terrain,” Science Robotics , vol. 8, no. 74, 2023

2023

Closest in time.