From Imitation to Refinement -- Residual RL for Precise Assembly

D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” nature , vol. 323, no. 6088, pp. 533–536, 1986

1986

Earlier work this paper cites.

D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” Advances in neural information processing systems , vol. 1, 1988

1988

Earlier work this paper cites.

S. Schaal, “Learning from Demonstration,” in Advances in Neural Information Processing Systems , vol. 9. MIT Press, 1996. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/1996/hash/68d13cf26c4b4f4f932e3eff990093ba-Abstract.html

1996

Earlier work this paper cites.

S. Schaal, “Learning from demonstration,” Advances in neural information processing systems , vol. 9, 1996

1996

Earlier work this paper cites.

——, “Is imitation learning the route to humanoid robots?” Trends in cognitive sciences , vol. 3, no. 6, pp. 233–242, 1999

1999

Earlier work this paper cites.

N. Ratliff, J. A. Bagnell, and S. S. Srinivasa, “Imitation learning for locomotion and manipulation,” in 2007 7th IEEE-RAS international conference on humanoid robots . IEEE, 2007, pp. 392–397

2007

Earlier work this paper cites.

S. Ross and D. Bagnell, “Efficient reductions for imitation learning,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics . JMLR Workshop and Conference Proceedings, 2010, pp. 661–668

2010

Earlier work this paper cites.

J. Kober and J. Peters, “Imitation and reinforcement learning,” IEEE Robotics & Automation Magazine , vol. 17, no. 2, pp. 55–62, 2010

2010

Earlier work this paper cites.

S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics . JMLR Workshop and Conference Proceedings, 2011, pp. 627–635

2011

Earlier work this paper cites.

S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics . JMLR Workshop and Conference Proceedings, 2011, pp. 627–635

2011

Earlier work this paper cites.

E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE, 2012, pp. 5026–5033

2012

Earlier work this paper cites.

A. M. Saxe, J. L. McClelland, and S. Ganguli, “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks,” arXiv preprint arXiv:1312.6120 , 2013

Original

2013

Earlier work this paper cites.

J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in International conference on machine learning . PMLR, 2015, pp. 1889–1897

2015

Earlier work this paper cites.

J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438 , 2015

Original

2015

Earlier work this paper cites.

F. Suárez-Ruiz and Q.-C. Pham, “A framework for fine robotic assembly,” in 2016 IEEE international conference on robotics and automation (ICRA) . IEEE, 2016, pp. 421–426

2016

Earlier work this paper cites.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

2016

Earlier work this paper cites.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” Aug. 2017, arXiv:1707.06347 [cs]. [Online]. Available: http://arxiv.org/abs/1707.06347

Original

2017

Earlier work this paper cites.

J. Schulman, X. Chen, and P. Abbeel, “Equivalence between policy gradients and soft q-learning,” arXiv preprint arXiv:1704.06440 , 2017

Original

2017

Earlier work this paper cites.

P. Agrawal, Computational sensorimotor learning . University of California, Berkeley, 2018

2018

Earlier work this paper cites.

T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” in 2018 IEEE international conference on robotics and automation (ICRA) . IEEE, 2018, pp. 5628–5635

2018

Earlier work this paper cites.

A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations,” in Proceedings of Robotics: Science and Systems (RSS) , 2018

2018

Earlier work this paper cites.

A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Overcoming exploration in reinforcement learning with demonstrations,” in 2018 IEEE international conference on robotics and automation (ICRA) . IEEE, 2018, pp. 6292–6299

2018

Earlier work this paper cites.

T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling, “Residual policy learning,” arXiv preprint arXiv:1812.06298 , 2018

Original

2018

Earlier work this paper cites.

A. Ajay, J. Wu, N. Fazeli, M. Bauza, L. P. Kaelbling, J. B. Tenenbaum, and A. Rodriguez, “Augmenting physical simulators with stochastic neural networks: Case study of planar pushing and bouncing,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2018, pp. 3066–3073

2018

Earlier work this paper cites.

A. Reuther, J. Kepner, C. Byun, S. Samsi, W. Arcand, D. Bestor, B. Bergeron, V. Gadepally, M. Houle, M. Hubbell, M. Jones, A. Klein, L. Milechin, J. Mullen, A. Prout, A. Rosa, C. Yee, and P. Michaleas, “Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis,” in 2018 IEEE High Performance extreme Computing Conference (HPEC) , Sep. 2018, pp. 1–6, iSSN: 2377-6943. [Online]. Available: https://ieeexplore.ieee.org/document/8547629

2018

Earlier work this paper cites.

T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine, “Residual reinforcement learning for robot control,” in 2019 international conference on robotics and automation (ICRA) . IEEE, 2019, pp. 6023–6029

2019

Earlier work this paper cites.

A. Galashov, S. M. Jayakumar, L. Hasenclever, D. Tirumala, J. Schwarz, G. Desjardins, W. M. Czarnecki, Y. W. Teh, R. Pascanu, and N. Heess, “Information asymmetry in kl-regularized rl,” arXiv preprint arXiv:1905.01240 , 2019

Original

2019

Earlier work this paper cites.

Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, “On the continuity of rotation representations in neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 5745–5753

2019

Earlier work this paper cites.

K. Kimble, K. Van Wyk, J. Falco, E. Messina, Y. Sun, M. Shibata, W. Uemura, and Y. Yokokohji, “Benchmarking protocols for evaluating small parts robotic assembly systems,” IEEE robotics and automation letters , vol. 5, no. 2, pp. 883–889, 2020

2020

Earlier work this paper cites.

A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine, “Parrot: Data-driven behavioral priors for reinforcement learning,” arXiv preprint arXiv:2011.10024 , 2020

Original

2020

Earlier work this paper cites.

J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” Science robotics , vol. 5, no. 47, p. eabc5986, 2020

2020

Earlier work this paper cites.

A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,” Advances in Neural Information Processing Systems , vol. 33, pp. 1179–1191, 2020

2020

Earlier work this paper cites.

A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser, “Tossingbot: Learning to throw arbitrary objects with residual physics,” IEEE Transactions on Robotics , vol. 36, no. 4, pp. 1307–1319, 2020

2020

Earlier work this paper cites.

G. Schoettler, A. Nair, J. Luo, S. Bahl, J. A. Ojea, E. Solowjow, and S. Levine, “Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 5548–5555

2020

Earlier work this paper cites.

J. Levinson, C. Esteves, K. Chen, N. Snavely, A. Kanazawa, A. Rostamizadeh, and A. Makadia, “An analysis of svd for deep rotation estimation,” Advances in Neural Information Processing Systems , vol. 33, pp. 22 554–22 565, 2020

2020

Cited alongside, same era.

Y. Lee, E. S. Hu, and J. J. Lim, “Ikea furniture assembly environment for long-horizon complex manipulation tasks,” in 2021 ieee international conference on robotics and automation (icra) . IEEE, 2021, pp. 6343–6349

2021

Cited alongside, same era.

M. Alakuijala, G. Dulac-Arnold, J. Mairal, J. Ponce, and C. Schmid, “Residual reinforcement learning from demonstrations,” arXiv preprint arXiv:2106.08050 , 2021

Original

2021

Cited alongside, same era.

A. Kumar, Z. Fu, D. Pathak, and J. Malik, “Rma: Rapid motor adaptation for legged robots,” arXiv preprint arXiv:2107.04034 , 2021

Original

2021

Cited alongside, same era.

H. Hu, S. Mirchandani, and D. Sadigh, “Imitation bootstrapped reinforcement learning,” arXiv preprint arXiv:2311.02198 , 2023

Original

2023

Later among the works it cites.

R. Ramrakhya, D. Batra, E. Wijmans, and A. Das, “Pirlnav: Pretraining with imitation and rl finetuning for objectnav,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 17 896–17 906

2023

Later among the works it cites.

K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine, “Training diffusion models with reinforcement learning,” arXiv preprint arXiv:2305.13301 , 2023

Original

2023

Later among the works it cites.

S. Haldar, J. Pari, A. Rai, and L. Pinto, “Teach a robot to fish: Versatile imitation from one minute of demonstrations,” arXiv preprint arXiv:2303.01497 , 2023

Original

2023

Later among the works it cites.