2020

Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic

Ren, Yangang, Duan, Jingliang, Li, Shengbo Eben et al.

Understand

Reinforcement learning (RL) has achieved remarkable performance in numerous sequential decision making and control tasks.

However, a common problem is that learned nearly optimal policy always overfits to the training environment and may not be extended to situations never encountered during training.
For practical applications, the randomness of environment usually leads to some devastating events, which should be the focus of safety-critical systems such as autonomous driving.
In this paper, we introduce the minimax formulation and distributional framework to improve the generalization ability of RL algorithms and develop the Minimax Distributional Soft Actor-Critic (Minimax DSAC) algorithm.

Robust reinforcement learning
Jun Morimoto and Kenji Doya · 2005
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy · 2015
Earlier work this paper cites.
Optimizing the cvar via sampling
Aviv Tamar, Yonatan Glassner, and Shie Mannor · 2015
Earlier work this paper cites.
A comprehensive survey on safe reinforcement learning
Javier Garcıa and Fernando Fernández · 2015
Earlier work this paper cites.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Earlier work this paper cites.
Epopt: Learning robust neural network policies using model ensembles
Original
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine · 2016
Earlier work this paper cites.

Adversarially robust policy learning: Active construction of physically-plausible perturbations
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese · 2017
Cited alongside, same era.
Robust adversarial reinforcement learning
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta · 2017
Cited alongside, same era.
Assessing generalization in deep reinforcement learning
Original
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song · 2018
Cited alongside, same era.
Sim-to-real transfer of robotic control with dynamics randomization
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel · 2018
Cited alongside, same era.
Robust deep reinforcement learning with adversarial attacks
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary · 2018
Cited alongside, same era.
Risk-sensitive reinforcement learning: A constrained optimization viewpoint
Original
A Prashanth L and Michael Fu · 2018
Cited alongside, same era.

Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Original
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine · 2018
Later among the works it cites.
Investigating generalisation in continuous deep reinforcement learning
Original
Chenyang Zhao, Olivier Siguad, Freek Stulp, and Timothy M Hospedales · 2019
Later among the works it cites.
Risk averse robust adversarial reinforcement learning
Original
Xinlei Pan, Daniel Seita, Yang Gao, and John Canny · 2019
Later among the works it cites.
Hierarchical reinforcement learning for self-driving decision-making without reliance on labelled driving data
Jingliang Duan, Shengbo Eben Li, Yang Guan, Qi Sun, and Bo Cheng · 2020
Closest in time.
Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors
Original
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, and Bo Cheng · 2020
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…