Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

Simple statistical gradient-following algorithms for connectionist reinforcement learning

Ronald J. Williams · 1992

Earlier work this paper cites.

Policy gradient methods for reinforcement learning with function approximation

Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour · 1999

Earlier work this paper cites.

A natural policy gradient

Sham M. Kakade · 2001

Earlier work this paper cites.

Relative entropy policy search

Jan Peters, Katharina Mülling, and Yasemin Altun · 2010

Earlier work this paper cites.

Playing atari with deep reinforcement learning

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller · 2013

Earlier work this paper cites.

Adam: A method for stochastic optimization

Original

Diederik P. Kingma and Jimmy Ba · 2014

Earlier work this paper cites.

Deep reinforcement learning through policy optimization

Pieter Abbeel and John Schulman · 2016

Cited alongside, same era.

Openai baselines

Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov · 2017

Cited alongside, same era.

Deep reinforcement learning that matters

Original

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger · 2017

Cited alongside, same era.

Towards generalization and simplicity in continuous control

Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M. Kakade · 2017

Cited alongside, same era.

Proximal policy optimization algorithms

Original

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017

Cited alongside, same era.

Where did my optimum go?: An empirical analysis of gradient descent optimization in policy gradient methods, 2018

Peter Henderson, Joshua Romoff, and Joelle Pineau · 2018

Later among the works it cites.

Simple random search provides a competitive approach to reinforcement learning

Original

Horia Mania, Aurelia Guy, and Benjamin Recht · 2018

Later among the works it cites.

Openai five

OpenAI · 2018

Later among the works it cites.

The mirage of action-dependent baselines in reinforcement learning

George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E. Turner, Zoubin Ghahramani, and Sergey Levine · 2018

Later among the works it cites.

Natural environment benchmarks for reinforcement learning, 2018

Amy Zhang, Yuxin Wu, and Joelle Pineau · 2018

Later among the works it cites.