2020

What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

Andrychowicz, Marcin, Raichuk, Anton, Stańczyk, Piotr et al.

Understand

In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks.

While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents.
Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations.
This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20].

Built on

“Asynchronous methods for deep reinforcement learning”
Volodymyr Mnih et al · 1937
Earlier work this paper cites.
“Robust estimation of a location parameter”
Peter Huber · 1992
Earlier work this paper cites.
“Simple statistical gradient-following algorithms for connectionist reinforcement learning”
Ronald Williams · 1992
Earlier work this paper cites.
“Introduction to reinforcement learning”
Richard Sutton and Andrew Barto · 1998
Earlier work this paper cites.
“Modeling purposeful adaptive behavior with the principle of maximum causal entropy”, 2010
Brian Ziebart · 2010
Earlier work this paper cites.
“Mujoco: A physics engine for model-based control”
Emanuel Todorov, Tom Erez and Yuval Tassa · 2012
Earlier work this paper cites.
“Playing atari with deep reinforcement learning”
Original
Volodymyr Mnih et al · 2013
Earlier work this paper cites.
“Delving deep into rectifiers: Surpassing human-level performance on imagenet classification”
Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun · 2015
Earlier work this paper cites.
“Adam: A Method for Stochastic Optimization”
Original
Diederik. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
“High-dimensional continuous control using generalized advantage estimation”
Original
John Schulman et al · 2015
Earlier work this paper cites.
“Trust region policy optimization”
John Schulman et al · 2015
Earlier work this paper cites.
“Openai gym”
Original
Greg Brockman et al · 2016
Earlier work this paper cites.

Similar

“Continuous control with deep reinforcement learning”
Timothy Lillicrap et al · 2016
Cited alongside, same era.
“Reproducibility of benchmarked deep reinforcement learning tasks for continuous control”
Original
Riashat Islam, Peter Henderson, Maziar Gomrokchi and Doina Precup · 2017
Cited alongside, same era.
“Time limits in reinforcement learning”
Original
Fabio Pardo, Arash Tavakoli, Vitaly Levdik and Petar Kormushev · 2017
Cited alongside, same era.
“Proximal policy optimization algorithms”
Original
John Schulman et al · 2017
Cited alongside, same era.
“Maximum a posteriori policy optimisation”
Original
Abbas Abdolmaleki et al · 2018
Cited alongside, same era.
“IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures”
Lasse Espeholt et al · 2018
Cited alongside, same era.
“Soft actor-critic algorithms and applications”
Original
Tuomas Haarnoja et al · 2018
Cited alongside, same era.
“Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine · 2018
Cited alongside, same era.
“Deep reinforcement learning that matters”
Peter Henderson et al · 2018
Cited alongside, same era.
“Challenging common assumptions in the unsupervised learning of disentangled representations”
Original
Francesco Locatello et al · 2018
Cited alongside, same era.
“Are gans created equal? a large-scale study”
Mario Lucic et al · 2018
Cited alongside, same era.
“Learning dexterous in-hand manipulation”
Original
M OpenAI et al · 2018
Cited alongside, same era.

Then

“The mirage of action-dependent baselines in reinforcement learning”
Original
George Tucker et al · 2018
Later among the works it cites.
“Implementation Matters in Deep RL: A Case Study on PPO and TRPO”
Logan Engstrom et al · 2019
Later among the works it cites.
“SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference”
Original
Lasse Espeholt et al · 2019
Later among the works it cites.
“When to trust your model: Model-based policy optimization”
Michael Janner, Justin Fu, Marvin Zhang and Sergey Levine · 2019
Later among the works it cites.
“Learning to drive in a day”
Alex Kendall et al · 2019
Later among the works it cites.
“Benchmarking model-based reinforcement learning”
Original
Eric Langlois et al · 2019
Later among the works it cites.
“Solving Rubik’s Cube with a Robot Hand”
Original
Ilge OpenAI et al · 2019
Later among the works it cites.
“Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning”
Original
Xue Peng, Aviral Kumar, Grace Zhang and Sergey Levine · 2019
Later among the works it cites.
“V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control”
Original
H Song et al · 2019
Later among the works it cites.
“Scaling laws for neural language models”
Original
Jared Kaplan et al · 2020
Closest in time.
“Designing Network Design Spaces”
Original
Ilija Radosavovic et al · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…