2018

Policy Optimization via Importance Sampling

Metelli, Alberto Maria, Papini, Matteo, Faccio, Francesco et al.

Understand

Policy optimization is an effective reinforcement learning approach to solve continuous control tasks.

Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse.
However, deciding when to stop optimizing and collect new trajectories is non-trivial, as it requires to account for the variance of the objective function estimate.
In this paper, we propose a novel, model-free, policy search algorithm, POIS, applicable in both action-based and parameter-based settings.

Built on

On measures of entropy and information
Alfréd Rényi · 1961
Earlier work this paper cites.
The convexity with respect to gaussian distributions of divergences of order α \alpha
Jacob Burbea · 1984
Earlier work this paper cites.
Advances in importance sampling
Timothy Classen Hesterberg · 1988
Earlier work this paper cites.
A note on importance sampling using standardized weights
Augustine Kong · 1992
Earlier work this paper cites.
Information and the accuracy attainable in the estimation of statistical parameters
C Radhakrishna Rao · 1992
Earlier work this paper cites.
Q-learning
Christopher JCH Watkins and Peter Dayan · 1992
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams · 1992
Earlier work this paper cites.
Incremental multi-step q-learning
Jing Peng and Ronald J Williams · 1994
Earlier work this paper cites.
Natural gradient works efficiently in learning
Shun-Ichi Amari · 1998
Earlier work this paper cites.
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto · 1998
Earlier work this paper cites.
The cross-entropy method for combinatorial and continuous optimization
Reuven Rubinstein · 1999
Earlier work this paper cites.
Actor-critic algorithms
Vijay R Konda and John N Tsitsiklis · 2000
Earlier work this paper cites.
Pegasus: A policy search method for large mdps and pomdps
Andrew Y Ng and Michael Jordan · 2000
Earlier work this paper cites.
Eligibility traces for off-policy policy evaluation
Doina Precup, Richard S Sutton, and Satinder P Singh · 2000
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour · 2000
Earlier work this paper cites.
Infinite-horizon policy-gradient estimation
Jonathan Baxter and Peter L Bartlett · 2001
Earlier work this paper cites.
Completely derandomized self-adaptation in evolution strategies
Nikolaus Hansen and Andreas Ostermeier · 2001
Earlier work this paper cites.
Approximately optimal approximate reinforcement learning
Sham Kakade and John Langford · 2002
Earlier work this paper cites.
A natural policy gradient
Sham M Kakade · 2002
Earlier work this paper cites.
Evolving neural networks through augmenting topologies
Kenneth O Stanley and Risto Miikkulainen · 2002
Earlier work this paper cites.
Stochastic policy gradient reinforcement learning on a simple 3d biped
Russ Tedrake, Teresa Weirui Zhang, and H Sebastian Seung · 2004
Earlier work this paper cites.
Learning tetris using the noisy cross-entropy method
István Szita and András Lörincz · 2006
Earlier work this paper cites.

Similar

Sampling techniques
William G Cochran · 2007
Cited alongside, same era.
Reinforcement learning by reward-weighted regression for operational space control
Jan Peters and Stefan Schaal · 2007
Cited alongside, same era.
Natural actor-critic
Jan Peters and Stefan Schaal · 2008
Cited alongside, same era.
Reinforcement learning of motor skills with policy gradients
Jan Peters and Stefan Schaal · 2008
Cited alongside, same era.
Policy gradients with parameter-based exploration for control
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber · 2008
Cited alongside, same era.
Natural evolution strategies
Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber · 2008
Cited alongside, same era.
Efficient natural evolution strategies
Yi Sun, Daan Wierstra, Tom Schaul, and Juergen Schmidhuber · 2009
Cited alongside, same era.
Learning bounds for importance weighting
Corinna Cortes, Yishay Mansour, and Mehryar Mohri · 2010
Cited alongside, same era.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Cited alongside, same era.
Adaptive step-size policy gradients with average reward metric
Takamitsu Matsubara, Tetsuro Morimura, and Jun Morimoto · 2010
Cited alongside, same era.
Natural policy gradient methods with parameter-based exploration for control tasks
Atsushi Miyamae, Yuichi Nagata, Isao Ono, and Shigenobu Kobayashi · 2010
Cited alongside, same era.
Relative entropy policy search
Jan Peters, Katharina Mülling, and Yasemin Altun · 2010
Cited alongside, same era.

Then

Efficient sample reuse in policy gradients with parameter-based exploration
Tingting Zhao, Hirotaka Hachiya, Voot Tangkaratt, Jun Morimoto, and Masashi Sugiyama · 2013
Later among the works it cites.
Markov decision processes: discrete stochastic dynamic programming
Martin L Puterman · 2014
Later among the works it cites.
Deterministic policy gradient algorithms
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller · 2014
Later among the works it cites.
Rényi divergence and kullback-leibler divergence
Tim Van Erven and Peter Harremos · 2014
Later among the works it cites.
Concentration inequalities for sums
Bernard Bercu, Bernard Delyon, and Emmanuel Rio · 2015
Later among the works it cites.
Continuous control with deep reinforcement learning
Original
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra · 2015
Later among the works it cites.
Trust region policy optimization
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz · 2015
Later among the works it cites.
High-dimensional continuous control using generalized advantage estimation
Original
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel · 2015
Later among the works it cites.
High confidence policy improvement
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh · 2015
Later among the works it cites.
High-confidence off-policy evaluation
Philip S Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh · 2015
Later among the works it cites.
Benchmarking deep reinforcement learning for continuous control
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel · 2016
Later among the works it cites.
Safe and efficient off-policy reinforcement learning
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare · 2016
Later among the works it cites.
Data-efficient off-policy policy evaluation for reinforcement learning
Philip Thomas and Emma Brunskill · 2016
Later among the works it cites.
Sample efficient actor-critic with experience replay
Original
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas · 2016
Later among the works it cites.
Importance sampling for fair policy selection
Shayan Doroudi, Philip S Thomas, and Emma Brunskill · 2017
Later among the works it cites.
Using options and covariance testing for long horizon off-policy policy evaluation
Zhaohan Guo, Philip S Thomas, and Emma Brunskill · 2017
Later among the works it cites.
Effective sample size for importance sampling based on discrepancy measures
Luca Martino, Víctor Elvira, and Francisco Louzada · 2017
Later among the works it cites.
Towards generalization and simplicity in continuous control
Aravind Rajeswaran, Kendall Lowrey, Emanuel V Todorov, and Sham M Kakade · 2017
Later among the works it cites.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Later among the works it cites.
The mirage of action-dependent baselines in reinforcement learning
Original
George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E Turner, Zoubin Ghahramani, and Sergey Levine · 2018
Closest in time.
Variance reduction for policy gradient with action-dependent factorized baselines
Original
Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…