2018

Horizon: Facebook's Open Source Applied Reinforcement Learning Platform

Gauci, Jason, Conti, Edoardo, Liang, Yitao et al.

Understand

In this paper we present Horizon, Facebook's open source applied reinforcement learning (RL) platform.

Horizon is an end-to-end platform designed to solve industry applied RL problems where datasets are large (millions to billions of observations), the feedback loop is slow (vs.
a simulator), and experiments must be done with care because they don't run in a simulator.
Unlike other RL platforms, which are often designed for fast prototyping and experimentation, Horizon is designed with production use cases as top of mind.

Built on

A generalization of sampling without replacement from a finite universe
Horvitz, D. G. and Thompson, D. J · 1952
Earlier work this paper cites.
A markovian decision process
Bellman, R · 1957
Earlier work this paper cites.
Mixture density networks
Bishop, C. M · 1994
Earlier work this paper cites.
Reinforcement learning: An introduction
Sutton, R. S., Barto, A. G., et al · 1998
Earlier work this paper cites.
Crafting papers on machine learning
Langley, P · 2000
Earlier work this paper cites.
The nonstochastic multiarmed bandit problem
Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E · 2002
Earlier work this paper cites.
Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions
Adomavicius, G. and Tuzhilin, A · 2005
Earlier work this paper cites.
The epoch-greedy algorithm for multi-armed bandits with side information
Langford, J. and Zhang, T · 2008
Earlier work this paper cites.
Spark: Cluster computing with working sets
Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., and Stoica, I · 2010
Earlier work this paper cites.
Pilco: A model-based and data-efficient approach to policy search
Deisenroth, M. and Rasmussen, C. E · 2011
Earlier work this paper cites.
Doubly robust policy evaluation and learning
Dudık, M., Langford, J., and Li, L · 2011
Earlier work this paper cites.
Counterfactual reasoning and learning systems: The example of computational advertising
Bottou, L., Peters, J., Quiñonero-Candela, J., Charles, D. X., Chickering, D. M., Portugaly, E., Ray, D., Simard, P., and Snelson, E · 2013
Earlier work this paper cites.
Caffe: Convolutional architecture for fast feature embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Deep reinforcement learning in large discrete action spaces
Original
Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Ioffe, S. and Szegedy, C · 2015
Earlier work this paper cites.

Similar

Continuous control with deep reinforcement learning
Original
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al · 2015
Cited alongside, same era.
Trust region policy optimization
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P · 2015
Cited alongside, same era.
A gaussian mixture model layer jointly optimized with discriminative features within a deep neural network architecture
Variani, E., McDermott, E., and Heigold, G · 2015
Cited alongside, same era.
Dueling network architectures for deep reinforcement learning
Original
Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., and De Freitas, N · 2015
Cited alongside, same era.
Embed to control: A locally linear latent dynamics model for control from raw images
Watter, M., Springenberg, J., Boedecker, J., and Riedmiller, M · 2015
Cited alongside, same era.
Making contextual decisions with low technical debt
Original
Agarwal, A., Bird, S., Cozowicz, M., Hoang, L., Langford, J., Lee, S., Li, J., Melamed, D., Oshri, G., Ribas, O., et al · 2016
Cited alongside, same era.
Openai gym
Original
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W · 2016
Cited alongside, same era.
Doubly robust off-policy value evaluation for reinforcement learning
Jiang, N. and Li, L · 2016
Cited alongside, same era.
Data-efficient off-policy policy evaluation for reinforcement learning
Thomas, P. and Brunskill, E · 2016
Cited alongside, same era.
Deep reinforcement learning with double q-learning
Van Hasselt, H., Guez, A., and Silver, D · 2016
Cited alongside, same era.
A distributional perspective on reinforcement learning
Original
Bellemare, M. G., Dabney, W., and Munos, R · 2017
Cited alongside, same era.

Then

Rainbow: Combining improvements in deep reinforcement learning
Original
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D · 2017
Later among the works it cites.
Ray rllib: A composable and scalable reinforcement learning library
Original
Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Gonzalez, J., Goldberg, K., and Stoica, I · 2017
Later among the works it cites.
Device placement optimization with reinforcement learning
Mirhoseini, A., Pham, H., Le, Q. V., Steiner, B., Larsen, R., Zhou, Y., Kumar, N., Norouzi, M., Bengio, S., and Dean, J · 2017
Later among the works it cites.
Pytorch, 2017
Paszke, A., Gross, S., Chintala, S., and Chanan, G · 2017
Later among the works it cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Later among the works it cites.
Optimal and adaptive off-policy evaluation in contextual bandits
Wang, Y.-X., Agarwal, A., and Dudik, M · 2017
Later among the works it cites.
Dopamine
Bellemare, M., Castro, P. S., Gelada, C., Kumar, S., and Moitra, S · 2018
Closest in time.
Onnx github repository, 2018
Exchange, O. N. N · 2018
Closest in time.
Off-policy deep reinforcement learning without exploration
Original
Fujimoto, S., Meger, D., and Precup, D · 2018
Closest in time.
World models
Original
Ha, D. and Schmidhuber, J · 2018
Closest in time.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Original
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S · 2018
Closest in time.
Tensorboardx
Huang, T.-W · 2018
Closest in time.
Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning
Nagabandi, A., Kahn, G., Fearing, R. S., and Levine, S · 2018
Closest in time.
Drn: A deep reinforcement learning framework for news recommendation
Zheng, G., Zhang, F., Zheng, Z., Xiang, Y., Yuan, N. J., Xie, X., and Li, Z · 2018
Closest in time.
Top-k off-policy correction for a reinforce recommender system
Chen, M., Beutel, A., Covington, P., Jain, S., Belletti, F., and Chi, E. H · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…