2020

A Self-Tuning Actor-Critic Algorithm

Zahavy, Tom, Xu, Zhongwen, Veeriah, Vivek et al.

Understand

Reinforcement learning algorithms are highly sensitive to the choice of hyperparameters, typically requiring significant manual effort to identify hyperparameters that perform well on a new domain.

In this paper, we take a step towards addressing this issue by using metagradients to automatically adapt hyperparameters online by meta-gradient descent (Xu et al., 2018).
We apply our algorithm, Self-Tuning Actor-Critic (STAC), to self-tune all the differentiable hyperparameters of an actor-critic loss function, to discover auxiliary tasks, and to improve off-policy learning using a novel leaky V-trace operator.
STAC is simple to use, sample efficient and does not require a significant increase in compute.

Built on

Adapting bias by gradient descent: An incremental version of delta-bar-delta
Sutton, R. S · 1992
Earlier work this paper cites.
Random search for hyper-parameter optimization
Bergstra, J. and Bengio, Y · 2012
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M · 2013
Earlier work this paper cites.
Rectifier nonlinearities improve neural network acoustic models
Maas, A. L., Hannun, A. Y., and Ng, A. Y · 2013
Earlier work this paper cites.
Empirical evaluation of rectified activations in convolutional network
Original
Xu, B., Wang, N., Chen, T., and Li, M · 2015
Earlier work this paper cites.
Reinforcement learning with unsupervised auxiliary tasks
Original
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K · 2016
Earlier work this paper cites.
Adaptive lambda least-squares temporal difference learning
Original
Mann, T. A., Penedones, H., Mannor, S., and Hester, T · 2016
Earlier work this paper cites.
A greedy approach to adapting the trace parameter for temporal difference learning
White, M. and White, A · 2016
Earlier work this paper cites.
Model-agnostic meta-learning for fast adaptation of deep networks
Finn, C., Abbeel, P., and Levine, S · 2017
Earlier work this paper cites.
Population based training of neural networks
Original
Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., et al · 2017
Earlier work this paper cites.

Similar

In-datacenter performance analysis of a tensor processing unit
Original
Jouppi, N. P., Young, C., Patil, N., Patterson, D. A., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., Boyle, R., Cantin, P., Chao, C., Clark, C., Coriell, J., Daley, M., Dau, M., Dean, J., Gelb, B., Ghaemmaghami, T. V., Gottipati, R., Gulland, W., Hagmann, R., Ho, R. C., Hogberg, D., Hu, J., Hundt, R., Hurt, D., Ibarz, J., Jaffey, A., Jaworski, A., Kaplan, A., Khaitan, H., Koch, A., Kumar, N., Lacy, S., Laudon, J., Law, J., Le, D., Leary, C., Liu, Z., Lucke, K., Lundin, A., MacKean, G., Maggiore, A., Mahony, M., Miller, K., Nagarajan, R., Narayanaswami, R., Ni, R., Nix, K., Norrie, T., Omernick, M., Penukonda, N., Phelps, A., Ross, J., Salek, A., Samadiani, E., Severn, C., Sizikov, G., Snelham, M., Souter, J., Steinberg, D., Swing, A., Tan, M., Thorson, G., Tian, B., Toma, H., Tuttle, E., Vasudevan, V., Walter, R., Wang, W., Wilcox, E., and Yoon, D. H · 2017
Cited alongside, same era.
JAX: composable transformations of Python+NumPy programs, 2018
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., and Wanderman-Milne, S · 2018
Cited alongside, same era.
Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures
Original
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al · 2018
Cited alongside, same era.
Soft actor-critic algorithms and applications
Original
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al · 2018
Cited alongside, same era.
Deepmind control suite
Original
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al · 2018
Cited alongside, same era.
Meta-gradient reinforcement learning
Xu, Z., van Hasselt, H. P., and Silver, D · 2018
Cited alongside, same era.
Hyperbolic discounting and learning over multiple horizons
Original
Fedus, W., Gelada, C., Bengio, Y., Bellemare, M. G., and Larochelle, H · 2019
Cited alongside, same era.
Fast efficient hyperparameter tuning for policy gradient methods
Paul, S., Kurin, V., and Whiteson, S · 2019
Cited alongside, same era.
Adaptive trade-offs in off-policy learning
Original
Rowland, M., Dabney, W., and Munos, R · 2019
Cited alongside, same era.
Adapting behaviour for learning progress
Original
Schaul, T., Borsa, D., Ding, D., Szepesvari, D., Ostrovski, G., Dabney, W., and Osindero, S · 2019
Cited alongside, same era.

Then

Off-policy actor-critic with shared experience replay
Original
Schmitt, S., Hessel, M., and Simonyan, K · 2019
Later among the works it cites.
Mastering atari, go, chess and shogi by planning with a learned model
Original
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al · 2019
Later among the works it cites.
Discovery of useful questions as auxiliary tasks
Veeriah, V., Hessel, M., Xu, Z., Rajendran, J., Lewis, R. L., Oh, J., van Hasselt, H. P., Silver, D., and Singh, S · 2019
Later among the works it cites.
RLax: Reinforcement Learning in JAX, 2020
Budden, D., Hessel, M., Quan, J., Kapturowski, S., Baumli, K., Bhupatiraju, S., Guy, A., and King, M · 2020
Closest in time.
An empirical investigation of the challenges of real-world reinforcement learning
Original
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., and Hester, T · 2020
Closest in time.
Haiku: Sonnet for JAX, 2020
Hennigan, T., Cai, T., Norman, T., and Babuschkin, I · 2020
Closest in time.
Optax: composable gradient transformation and optimisation, in jax!, 2020
Hessel, M., Budden, D., Viola, F., Rosca, M., Sezener, E., and Hennigan, T · 2020
Closest in time.
Online hyper-parameter tuning in off-policy learning via evolutionary strategies
Original
Tang, Y. and Choromanski, K · 2020
Closest in time.
On learning intrinsic rewards for policy gradient methods
Zheng, Z., Oh, J., and Singh, S · 2020
Closest in time.
Podracer architectures for scalable reinforcement learning, 2021
Hessel, M., Kroiss, M., Clark, A., Kemaev, I., Quan, J., Keck, T., Viola, F., and van Hasselt, H · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…