Fetching the paper…

Reading the bibliography…

2023

Spreeze: High-Throughput Parallel Reinforcement Learning Framework

Hou, Jing, Chen, Guang, Zhang, Ruiqi et al.

Original
PDF
Post on X

Understand

The promotion of large-scale applications of reinforcement learning (RL) requires efficient training computation.

While existing parallel RL frameworks encompass a variety of RL algorithms and parallelization techniques, the excessively burdensome communication frameworks hinder the attainment of the hardware's limit for final throughput and training effects on a single desktop.
In this paper, we propose Spreeze, a lightweight parallel framework for RL that efficiently utilizes a single desktop hardware resource to approach the throughput limit.
We asynchronously parallelize the experience sampling, network update, performance evaluation, and visualization operations, and employ multiple efficient data transmission techniques to transfer various types of data between processes.

Spreeze: High-Throughput Parallel Reinforcement Learning Framework · Around

Built on

J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, et al
2012
Earlier work this paper cites.
T. Komoda, S. Hayashi, T. Nakada, S. Miwa, and H. Nakamura, “Power capping of cpu-gpu heterogeneous systems through coordinating dvfs and task mapping,” in 2013 IEEE 31st International Conference on computer design (ICCD)
2013
Earlier work this paper cites.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971
Original
2015
Earlier work this paper cites.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in International conference on machine learning
2016
Earlier work this paper cites.
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al
2016
Earlier work this paper cites.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv:1606.01540
Original
2016
Earlier work this paper cites.
E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” URL http://pybullet. org
2016
Earlier work this paper cites.
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al
2017
Earlier work this paper cites.
N. Levine, T. Zahavy, D. J. Mankowitz, A. Tamar, and S. Mannor, “Shallow updates for deep reinforcement learning,” arXiv preprint arXiv:1705.07461
Original
2017
Earlier work this paper cites.
E. Hoffer, I. Hubara, and D. Soudry, “Train longer, generalize better: closing the generalization gap in large batch training of neural networks,” arXiv preprint arXiv:1705.08741
Original
2017
Earlier work this paper cites.

Similar

M. Papini, M. Pirotta, and M. Restelli, “Adaptive batch size for safe policy gradients,” in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS)
2017
Cited alongside, same era.
S. Han and Y. Sung, “Amber: Adaptive multi-batch experience replay for continuous action control,” arXiv preprint arXiv:1710.04423
Original
2017
Cited alongside, same era.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347
Original
2017
Cited alongside, same era.
E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. Gonzalez, M. Jordan, and I. Stoica, “Rllib: Abstractions for distributed reinforcement learning,” in International Conference on Machine Learning
2018
Cited alongside, same era.

Then

G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, and T. Lillicrap, “Distributed distributional deterministic policy gradients,” arXiv preprint arXiv:1804.08617
Original
2018
Later among the works it cites.
S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in ICML
2018
Later among the works it cites.
K. M. Abughalieh and S. G. Alawneh, “A survey of parallel implementations for model predictive control,” IEEE Access
2019
Later among the works it cites.
A. Stooke and P. Abbeel, “rlpyt: A research code base for deep reinforcement learning in pytorch,” arXiv preprint arXiv:1909.01500
Original
2019
Later among the works it cites.
E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, “Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames,” arXiv preprint arXiv:1911.00357

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. Van Hasselt, and D. Silver, “Distributed prioritized experience replay,” arXiv preprint arXiv:1803.00933

2018

Cited alongside, same era.

L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al

2018

Cited alongside, same era.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in International conference on machine learning

2018

Cited alongside, same era.

A. Stooke and P. Abbeel, “Accelerated methods for deep reinforcement learning,” arXiv preprint arXiv:1803.02811

2018

Cited alongside, same era.

C. Ying, S. Kumar, D. Chen, T. Wang, and Y. Cheng, “Image classification at supercomputer scale,” arXiv preprint arXiv:1811.06992

2018

Cited alongside, same era.

S. Ahn, J. Kim, E. Lim, and S. Kang, “Soft memory box: A virtual shared memory framework for fast deep neural network training in distributed high performance computing,” IEEE Access

2018

Cited alongside, same era.

P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, et al

Cited in the paper.

2019

Later among the works it cites.

L. Espeholt, R. Marinier, P. Stanczyk, K. Wang, and M. Michalski, “Seed rl: Scalable and efficient deep-rl with accelerated central inference,” arXiv preprint arXiv:1910.06591

Original

2019

Later among the works it cites.

Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, et al

2019

Later among the works it cites.

M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, F. Yang, K. Baumli, et al

2020

Later among the works it cites.

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al

2020

Later among the works it cites.

R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare, “Deep reinforcement learning at the edge of the statistical precipice,” arXiv preprint arXiv:2108.13264

2021

Later among the works it cites.