2020

Zeroth-Order Supervised Policy Improvement

Sun, Hao, Xu, Ziping, Song, Yuhang et al.

Understand

Policy gradient (PG) algorithms have been widely used in reinforcement learning (RL).

However, PG algorithms rely on exploiting the value function being learned with the first-order update locally, which results in limited sample efficiency.
In this work, we propose an alternative method called Zeroth-Order Supervised Policy Improvement (ZOSPI).
ZOSPI exploits the estimated value function $Q$ globally while preserving the local exploitation of the PG methods based on zeroth-order policy optimization.

Built on

R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning
1992
Earlier work this paper cites.
C. M. Bishop, “Mixture density networks,” 1994
1994
Earlier work this paper cites.
R. S. Sutton and A. G. Barto, “Reinforcement learning: An introduction,” 1998
1998
Earlier work this paper cites.
R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Advances in neural information processing systems
2000
Earlier work this paper cites.
V. R. Konda and J. N. Tsitsiklis, “Actor-critic algorithms,” in Advances in neural information processing systems
2000
Earlier work this paper cites.
R. I. Brafman and M. Tennenholtz, “R-max-a general polynomial time algorithm for near-optimal reinforcement learning,” Journal of Machine Learning Research
2002
Earlier work this paper cites.
M. Kuss and C. E. Rasmussen, “Gaussian processes in reinforcement learning,” in Advances in neural information processing systems
2004
Earlier work this paper cites.
Y. Engel, S. Mannor, and R. Meir, “Reinforcement learning with gaussian processes,” in Proceedings of the 22nd international conference on Machine learning
2005
Earlier work this paper cites.
PhD thesis, echnische Universität Darmstadt Darmstadt, Germany, 2006
M. Kuss, Gaussian process models for robust regression, classification, and reinforcement learning · 2006
Earlier work this paper cites.
J. Peters and S. Schaal, “Natural actor-critic,” Neurocomputing
2008
Earlier work this paper cites.
T. Jaksch, R. Ortner, and P. Auer, “Near-optimal regret bounds for reinforcement learning,” Journal of Machine Learning Research
2010
Earlier work this paper cites.
S. Levine, Z. Popovic, and V. Koltun, “Nonlinear inverse reinforcement learning with gaussian processes,” in Advances in Neural Information Processing Systems
2011
Earlier work this paper cites.
T. Degris, M. White, and R. S. Sutton, “Off-policy actor-critic,” arXiv preprint arXiv:1205.4839
Original
2012
Earlier work this paper cites.
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control.,” in IROS
2012
Earlier work this paper cites.
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in ICML
2014
Earlier work this paper cites.
R. Grande, T. Walsh, and J. How, “Sample efficient reinforcement learning with gaussian processes,” in International Conference on Machine Learning
2014
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al
2015
Earlier work this paper cites.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971
Original
2015
Earlier work this paper cites.
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in International conference on machine learning
2015
Earlier work this paper cites.

Similar

S. Gu, T. Lillicrap, Z. Ghahramani, R. E. Turner, and S. Levine, “Q-prop: Sample-efficient policy gradient with an off-policy critic,” arXiv preprint arXiv:1611.02247
Original
2016
Cited alongside, same era.
Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas, “Sample efficient actor-critic with experience replay,” arXiv preprint arXiv:1611.01224
Original
2016
Cited alongside, same era.
N. Usunier, G. Synnaeve, Z. Lin, and S. Chintala, “Episodic exploration for deep deterministic policies for starcraft micromanagement,” 2016
2016
Cited alongside, same era.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016
2016
Cited alongside, same era.
I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration via bootstrapped dqn,” in Advances in neural information processing systems
2016
Cited alongside, same era.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347
Original
2017
Cited alongside, same era.
T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70
2017
Cited alongside, same era.
T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever, “Evolution strategies as a scalable alternative to reinforcement learning,” arXiv preprint arXiv:1703.03864
Original
2017
Cited alongside, same era.
Y. Wang, S. Du, S. Balakrishnan, and A. Singh, “Stochastic zeroth-order optimization in high dimensions,” arXiv preprint arXiv:1710.10551
Original
2017
Cited alongside, same era.
P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y. Wu, and P. Zhokhov, “Openai baselines.” https://github.com/openai/baselines , 2017
2017
Cited alongside, same era.
M. G. Azar, I. Osband, and R. Munos, “Minimax regret bounds for reinforcement learning,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70
2017
Cited alongside, same era.
J. Pachocki, G. Brockman, J. Raiman, S. Zhang, H. Pondé, J. Tang, F. Wolski, C. Dennison, R. Jozefowicz, P. Debiak, et al
2018
Cited alongside, same era.

Then

S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” arXiv preprint arXiv:1812.02900
Original
2018
Later among the works it cites.
C. Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan, “Is q-learning provably efficient?,” in Advances in Neural Information Processing Systems
2018
Later among the works it cites.
I. Osband, J. Aslanides, and A. Cassirer, “Randomized prior functions for deep reinforcement learning,” in Advances in Neural Information Processing Systems
2018
Later among the works it cites.
Y. Fan, L. Chen, and Y. Wang, “Efficient model-free reinforcement learning using gaussian process,” arXiv preprint arXiv:1812.04359
Original
2018
Later among the works it cites.
O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al
2019
Later among the works it cites.
S. Zhang, W. Boehmer, and S. Whiteson, “Generalized off-policy actor-critic,” arXiv preprint arXiv:1903.11329
Original
2019
Later among the works it cites.
K. Ciosek, Q. Vuong, R. Loftin, and K. Hofmann, “Better exploration with optimistic actor critic,” in Advances in Neural Information Processing Systems
2019
Later among the works it cites.
C. Tessler, G. Tennenholtz, and S. Mannor, “Distributional policy optimization: An alternative approach for continuous control,” arXiv preprint arXiv:1905.09855
Original
2019
Later among the works it cites.
H. Sun, Z. Li, X. Liu, B. Zhou, and D. Lin, “Policy continuation with hindsight inverse dynamics,” in Advances in Neural Information Processing Systems
2019
Later among the works it cites.
D. Ghosh, A. Gupta, J. Fu, A. Reddy, C. Devine, B. Eysenbach, and S. Levine, “Learning to reach goals without reinforcement learning,” arXiv preprint arXiv:1912.06088
Original
2019
Later among the works it cites.
C. Zhang, Y. Li, and J. Li, “Policy search by target distribution learning for continuous control,” arXiv preprint arXiv:1905.11041
Original
2019
Later among the works it cites.
H. F. Song, A. Abdolmaleki, J. T. Springenberg, A. Clark, H. Soyer, J. W. Rae, S. Noury, A. Ahuja, S. Liu, D. Tirumala, et al
Original
2019
Later among the works it cites.
R. Simmons-Edler, B. Eisner, E. Mitchell, S. Seung, and D. Lee, “Q-learning for continuous actions with cross-entropy guided policies,” arXiv preprint arXiv:1903.10605
Original
2019
Later among the works it cites.
D. Golovin, J. Karro, G. Kochanski, C. Lee, X. Song, et al
Original
2019
Later among the works it cites.
E.-V. Vlatakis-Gkaragkounis, L. Flokas, and G. Piliouras, “Efficiently avoiding saddle points with zero order methods: No gradients required,” in Advances in Neural Information Processing Systems
2019
Later among the works it cites.
R. Agarwal, D. Schuurmans, and M. Norouzi, “Striving for simplicity in off-policy deep reinforcement learning,” arXiv preprint arXiv:1907.04543
Original
2019
Later among the works it cites.
A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine, “Stabilizing off-policy q-learning via bootstrapping error reduction,” in Advances in Neural Information Processing Systems
2019
Later among the works it cites.
Q. Bai, M. Agarwal, and V. Aggarwal, “Escaping saddle points for zeroth-order non-convex optimization using estimated gradient descent,” in 2020 54th Annual Conference on Information Sciences and Systems (CISS)
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…