2021

Training Larger Networks for Deep Reinforcement Learning

Ota, Kei, Jha, Devesh K., Kanezaki, Asako

Understand

The success of deep learning in the computer vision and natural language processing communities can be attributed to training of very deep neural networks with millions or billions of parameters which can then be trained with massive amounts of data.

However, similar trend has largely eluded training of deep reinforcement learning (RL) algorithms where larger networks do not lead to performance improvement.
Previous work has shown that this is mostly due to instability during training of deep RL agents when using larger networks.
In this paper, we make an attempt to understand and address training of larger networks for deep RL.

Built on

Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., and Frasconi, P · 1994
Earlier work this paper cites.
Pattern Recognition and Machine Learning (Information Science and Statistics)
Bishop, C. M · 2006
Earlier work this paper cites.
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H., et al · 2007
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L., Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Todorov, E., Erez, T., and Tassa, Y · 2012
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Ioffe, S. and Szegedy, C · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Learning state representation for deep actor-critic control
Munk, J., Kober, J., and Babuška, R · 2016
Earlier work this paper cites.
Prioritized experience replay
Schaul, T., Quan, J., Antonoglou, I., and Silver, D · 2016
Earlier work this paper cites.
Deep reinforcement learning with double q-learning
Van Hasselt, H., Guez, A., and Silver, D · 2016
Earlier work this paper cites.
Densely connected convolutional networks
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q · 2017
Earlier work this paper cites.
Reinforcement learning with unsupervised auxiliary tasks
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K · 2017
Earlier work this paper cites.
The expressive power of neural networks: a view from the width
Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L · 2017
Earlier work this paper cites.
The loss surface of deep and wide neural networks
Nguyen, Q. and Hein, M · 2017
Earlier work this paper cites.

Similar

Searching for Activation Functions
Original
Ramachandran, P., Zoph, B., and Le, Q. V · 2017
Cited alongside, same era.
New modeling of multilayer perceptron architecture optimization with regularization: an application to pattern classification
Ramchoun, H., Idrissi, M. A. J., Ghanou, Y., and Ettaouil, M · 2017
Cited alongside, same era.
Loss is its own reward: Self-supervision for reinforcement learning
Shelhamer, E., Mahmoudieh, P., Argus, M., and Darrell, T · 2017
Cited alongside, same era.
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O · 2017
Cited alongside, same era.
Addressing function approximation error in actor-critic methods
Fujimoto, S., van Hoof, H., and Meger, D · 2018
Cited alongside, same era.
Recurrent world models facilitate policy evolution
Ha, D. and Schmidhuber, J · 2018
Cited alongside, same era.
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S · 2018
Cited alongside, same era.
Deep reinforcement learning that matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D · 2018
Cited alongside, same era.
Rainbow: Combining improvements in deep reinforcement learning
Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M. G., and Silver, D · 2018
Cited alongside, same era.
Distributed prioritized experience replay
Original
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., Van Hasselt, H., and Silver, D · 2018
Cited alongside, same era.
State Representation Learning for Control: An Overview
Lesort, T., Díaz-Rodríguez, N., Goudou, J.-F., and Filliat, D · 2018
Cited alongside, same era.
Visualizing the loss landscape of neural nets
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T · 2018
Cited alongside, same era.

Then

Diagnosing bottlenecks in deep q-learning algorithms
Fu, J., Kumar, A., Soh, M., and Levine, S · 2019
Later among the works it cites.
Recurrent experience replay in distributed reinforcement learning
Kapturowski, S., Ostrovski, G., Dabney, W., Quan, J., and Munos, R · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I · 2019
Later among the works it cites.
Efficientnet: Rethinking model scaling for convolutional neural networks
Tan, M. and Le, Q · 2019
Later among the works it cites.
Wider or deeper: Revisiting the resnet model for visual recognition
Wu, Z., Shen, C., and Van Den Hengel, A · 2019
Later among the works it cites.
Language models are few-shot learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D · 2020
Later among the works it cites.
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G · 2020
Later among the works it cites.
Revisiting fundamentals of experience replay
Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., and Dabney, W · 2020
Later among the works it cites.
Momentum contrast for unsupervised visual representation learning
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R · 2020
Later among the works it cites.
Data-efficient image recognition with contrastive predictive coding
Henaff, O · 2020
Later among the works it cites.
Can increasing input dimensionality improve deep reinforcement learning?
Ota, K., Oiki, T., Jha, D., Mariyama, T., and Nikovski, D · 2020
Later among the works it cites.
D2rl: Deep dense architectures in reinforcement learning, 2020
Sinha, S., Bharadhwaj, H., Srinivas, A., and Garg, A · 2020
Later among the works it cites.
Decoupling representation learning from reinforcement learning, 2020
Stooke, A., Lee, K., Abbeel, P., and Laskin, M · 2020
Later among the works it cites.
What matters in on-policy reinforcement learning? a large-scale empirical study
Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., and Bachem, O · 2021
Closest in time.
Implicit under-parameterization inhibits data-efficient deep reinforcement learning
Aviral Kumar, Rishabh Agarwal, D. G. and Levine, S · 2021
Closest in time.
Scaling laws for transfer
Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S · 2021
Closest in time.
Regularization matters in policy optimization
Liu, Z., Li, X., Kang, B., and Darrell, T · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…