Fetching the paper…

Reading the bibliography…

2020

Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards

Chen, Zhixin, Lin, Mengxiang

Original
PDF
Post on X

Understand

The application of reinforcement learning (RL) in robotic control is still limited in the environments with sparse and delayed rewards.

In this paper, we propose a practical self-imitation learning method named Self-Imitation Learning with Constant Reward (SILCR).
Instead of requiring hand-defined immediate rewards from environments, our method assigns the immediate rewards at each timestep with constant values according to their final episodic rewards.
In this way, even if the dense rewards from environments are unavailable, every action taken by the agents would be guided properly.

Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards · Around

Built on

J. A. Bagnell and B. D. Ziebart, “Modeling purposeful adaptive behavior with the principle of maximum causal entropy,” 2010
2010
Earlier work this paper cites.
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 5026–5033, 2012
2012
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature , vol. 518, pp. 529–533, 2015
2015
Earlier work this paper cites.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” ArXiv , vol. abs/1606.01540, 2016
Original
2016
Earlier work this paper cites.
Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking deep reinforcement learning for continuous control,” ArXiv , vol. abs/1604.06778, 2016
Original
2016
Earlier work this paper cites.
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. R. Baker, M. Lai, A. Bolton, Y. Chen, T. P. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, “Mastering the game of go without human knowledge,” Nature , vol. 550, pp. 354–359, 2017
2017
Earlier work this paper cites.
M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. H. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight experience replay,” ArXiv , vol. abs/1707.01495, 2017
Original
2017
Earlier work this paper cites.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised prediction,” 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pp. 488–489, 2017
2017
Earlier work this paper cites.

Similar

J. Oh, Y. Guo, S. Singh, and H. Lee, “Self-imitation learning,” in ICML , 2018
2018
Cited alongside, same era.
Y. Guo, J. Oh, S. Singh, and H. Lee, “Generative adversarial self-imitation learning,” ArXiv , vol. abs/1812.00950, 2018
Original
2018
Cited alongside, same era.
T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “Soft actor-critic algorithms and applications,” ArXiv , vol. abs/1812.05905, 2018
Original
2018
Cited alongside, same era.
K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly, “The gan landscape: Losses, architectures, regularization, and normalization,” ArXiv , vol. abs/1807.04720, 2018
Original
2018
Cited alongside, same era.
J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Science Robotics , vol. 4, 2019

Then

Y. Guo, J. wook Choi, M. Moczulski, S. Bengio, M. Norouzi, and H. Lee, “Self-imitation learning via trajectory-conditioned policy for hard-exploration tasks,” arXiv: Learning , 2019
2019
Later among the works it cites.
Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” ArXiv , vol. abs/1810.12894, 2019
Original
2019
Later among the works it cites.
H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine, “The ingredients of real-world robotic reinforcement learning,” ArXiv , vol. abs/2004.12570, 2020
Original
2020
Closest in time.
T. Li, K. Srinivasan, M. Q.-H. Meng, W. Yuan, and J. Bohg, “Learning hierarchical control for robust in-hand manipulation,” 2020 IEEE International Conference on Robotics and Automation (ICRA) , pp. 8855–8862, 2020
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2019

Cited alongside, same era.

H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar, “Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost,” 2019 International Conference on Robotics and Automation (ICRA) , pp. 3651–3657, 2019

2019

Cited alongside, same era.

J. Choi, K. sik Park, M. Kim, and S. Seok, “Deep reinforcement learning of navigation in a complex and crowded environment with a limited field of view,” 2019 International Conference on Robotics and Automation (ICRA) , pp. 5993–6000, 2019

2019

Cited alongside, same era.

T. Gangwani, Q. Liu, and J. Peng, “Learning self-imitating diverse policies,” ArXiv , vol. abs/1805.10309, 2019

2019

Cited alongside, same era.

S. Reddy, A. D. Dragan, and S. Levine, “Sqil: Imitation learning via reinforcement learning with sparse rewards,” arXiv: Learning , 2020

2020

Closest in time.

A. Singh, E. Jang, A. Irpan, D. Kappler, M. Dalal, S. Levine, M. Khansari, and C. Finn, “Scalable multi-task imitation learning with autonomous improvement,” 2020 IEEE International Conference on Robotics and Automation (ICRA) , pp. 2167–2173, 2020

2020

Closest in time.

K. Brantley, W. Sun, and M. Henaff, “Disagreement-regularized imitation learning,” in ICLR , 2020

2020

Closest in time.