Fetching the paper…

Reading the bibliography…

2022

Unentangled quantum reinforcement learning agents in the OpenAI Gym

Hsiao, Jen-Yueh, Du, Yuxuan, Chiang, Wei-Yin et al.

Original
PDF
Post on X

Understand

Classical reinforcement learning (RL) has generated excellent results in different regions; however, its sample inefficiency remains a critical issue.

In this paper, we provide concrete numerical evidence that the sample efficiency (the speed of convergence) of quantum RL could be better than that of classical RL, and for achieving comparable learning performance, quantum RL could use much (at least one order of magnitude) fewer trainable parameters than classical RL.
Specifically, we employ the popular benchmarking environments of RL in the OpenAI Gym, and show that our quantum RL agent converges faster than classical fully-connected neural networks (FCNs) in the tasks of CartPole and Acrobot under the same optimization process.
We also successfully train the first quantum RL agent that can complete the task of LunarLander in the OpenAI Gym.

Unentangled quantum reinforcement learning agents in the OpenAI Gym · Around

Built on

J. Bhandari and D. Russo, Global optimality guarantees for policy gradient methods (2020), arXiv:1906.01786 [cs.LG]
Original
1906
Earlier work this paper cites.
A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan, On the theory of policy gradient methods: Optimality, approximation, and distribution shift (2020), arXiv:1908.00261 [cs.LG]
Original
1908
Earlier work this paper cites.
R. Bellman, Indiana Univ. Math. J. 6
1957
Earlier work this paper cites.
L. K. Grover, A fast quantum mechanical algorithm for database search (1996), arXiv:quant-ph/9605043 [quant-ph]
1996
Earlier work this paper cites.
R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning , 1st ed. (MIT Press, Cambridge, MA, USA, 1998)
1998
Earlier work this paper cites.
A. Ahuja and S. Kapoor, A quantum algorithm for finding the maximum (1999), arXiv:quant-ph/9911082 [quant-ph]
1999
Earlier work this paper cites.
S. S. Du, J. D. Lee, G. Mahajan, and R. Wang, Agnostic q-learning with function approximation in deterministic systems: Tight bounds on approximation error and sample complexity (2020a), arXiv:2002.07125 [cs.LG]
Original
2002
Earlier work this paper cites.
A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky, A. Pritzel, A. Bolt, and C. Blundell, Never give up: Learning directed exploration strategies (2020), arXiv:2002.06038 [cs.LG]
Original
2002
Earlier work this paper cites.
G. Liu, R. Wu, H.-T. Cheng, J. Wang, J. Ooi, L. Li, A. Li, W. L. S. Li, C. Boutilier, and E. Chi, Data efficient training for reinforcement learning with adaptive behavior policy sharing (2020), arXiv:2002.05229 [cs.LG]
Original
2002
Earlier work this paper cites.
A. Mirhoseini, A. Goldie, M. Yazgan, J. Jiang, E. Songhori, S. Wang, Y.-J. Lee, E. Johnson, O. Pathak, S. Bae, A. Nazi, J. Pak, A. Tong, K. Srinivasa, W. Hang, E. Tuncer, A. Babu, Q. V. Le, J. Laudon, R. Ho, R. Carpenter, and J. Dean, Chip placement with deep reinforcement learning (2020), arXiv:2004.10746 [cs.LG]
Original
2004
Earlier work this paper cites.
D. Dong, C. Chen, H. Li, and T.-J. Tarn, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 38
2008
Earlier work this paper cites.
V. Giovannetti, S. Lloyd, and L. Maccone, Phys. Rev. Lett. 100
2008
Earlier work this paper cites.
S. Y.-C. Chen, S. Yoo, and Y.-L. L. Fang, Quantum long short-term memory (2020c), arXiv:2009.01783 [quant-ph]
Original
2009
Earlier work this paper cites.
J. Zhang, J. Kim, B. O’Donoghue, and S. Boyd, Sample efficient reinforcement learning with reinforce (2020a), arXiv:2010.11364 [cs.LG]
Original
2010
Earlier work this paper cites.
Y. Du, T. Huang, S. You, M.-H. Hsieh, and D. Tao, Quantum circuit architecture search: error mitigation and trainability enhancement for variational quantum solvers (2020c), arXiv:2010.10217 [quant-ph]
Original
2010
Earlier work this paper cites.
D. N. Panou and M. Reczko, Deepfoldit – a deep reinforcement learning neural network folding proteins (2020), arXiv:2011.03442 [q-bio.BM]
Original
2011
Earlier work this paper cites.
S. Y.-C. Chen, C.-M. Huang, C.-W. Hsing, and Y.-J. Kao, Hybrid quantum-classical classifier based on tensor network and variational quantum circuit (2020a), arXiv:2011.14651 [quant-ph]
Original
2011
Earlier work this paper cites.
K. Zhang, M.-H. Hsieh, L. Liu, and D. Tao, Toward trainability of quantum neural networks (2020b), arXiv:2011.06258 [quant-ph]
Original
2011
Earlier work this paper cites.
S. Y.-C. Chen, T.-C. Wei, C. Zhang, H. Yu, and S. Yoo, Quantum convolutional neural networks for high energy physics data analysis (2020b), arXiv:2012.12177 [cs.LG]
Original
2012
Earlier work this paper cites.
M. S. Rudolph, N. B. Toussaint, A. Katabarwa, S. Johri, B. Peropadre, and A. Perdomo-Ortiz, Generation of high-resolution handwritten digits with an ion-trap quantum computer (2021), arXiv:2012.03924 [quant-ph]
Original
2012
Earlier work this paper cites.
G. D. Paparo, V. Dunjko, A. Makmal, M. A. Martin-Delgado, and H. J. Briegel, Physical Review X 4
2014
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, Nature 518
2015
Earlier work this paper cites.
A. Montanaro, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 471
2015
Earlier work this paper cites.
S. Aaronson (2015)
2015
Earlier work this paper cites.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, Openai gym (2016), arXiv:1606.01540
Original
2016
Earlier work this paper cites.
D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, Mastering chess and shogi by self-play with a general reinforcement learning algorithm (2017), arXiv:1712.01815 [cs.AI]
Original
2017
Earlier work this paper cites.
M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, Rainbow: Combining improvements in deep reinforcement learning (2017), arXiv:1710.02298 [cs.AI]
Original
2017
Earlier work this paper cites.

Similar

C. S. Hamilton, R. Kruse, L. Sansoni, S. Barkhofen, C. Silberhorn, and I. Jex, Phys. Rev. Lett. 119
2017
Cited alongside, same era.
J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, Nature 549
2017
Cited alongside, same era.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms (2017), arXiv:1707.06347 [cs.LG]
Original
2017
Cited alongside, same era.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (A Bradford Book, Cambridge, MA, USA, 2018)
2018
Cited alongside, same era.
A. Irpan, Deep reinforcement learning doesn’t work yet, https://www.alexirpan.com/2018/02/14/rl-hard.html (2018)
2018
Cited alongside, same era.

Then

N. A. Nghiem, S. Y.-C. Chen, and T.-C. Wei, Phys. Rev. Research 3
2021
Later among the works it cites.
C.-H. H. Yang, J. Qi, S. Y.-C. Chen, P.-Y. Chen, S. M. Siniscalchi, X. Ma, and C.-H. Lee, in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2021) pp. 6523–6527
2021
Later among the works it cites.
Y. Du and D. Tao, On exploring practical potentials of quantum auto-encoder with advantages (2021), arXiv:2106.15432 [quant-ph]
Original
2021
Later among the works it cites.
S. Jerbi, C. Gyurik, S. C. Marshall, H. J. Briegel, and V. Dunjko, Parametrized quantum policies for reinforcement learning (2021), arXiv:2103.05577 [quant-ph]
Original
2021
Later among the works it cites.
A. Skolik, S. Jerbi, and V. Dunjko, Quantum agents in the gym: a variational quantum algorithm for deep q-learning (2021), arXiv:2103.15084 [quant-ph]

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

O. Nachum, S. Gu, H. Lee, and S. Levine, Data-efficient hierarchical reinforcement learning (2018), arXiv:1805.08296 [cs.LG]

2018

Cited alongside, same era.

J. Preskill, Quantum 2

2018

Cited alongside, same era.

E. Farhi and H. Neven, arXiv preprint arXiv:1802.06002 (2018), arXiv:1802.06002

2018

Cited alongside, same era.

A. Khoshaman, W. Vinci, B. Denis, E. Andriyash, H. Sadeghi, and M. H. Amin, Quantum Science and Technology 4

2018

Cited alongside, same era.

K. Mitarai, M. Negoro, M. Kitagawa, and K. Fujii, Physical Review A 98

2018

Cited alongside, same era.

C. Ciliberto, M. Herbster, A. D. Ialongo, M. Pontil, A. Rocchetto, S. Severini, and L. Wossnig, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 474

2018

Cited alongside, same era.

J. R. McClean, S. Boixo, V. N. Smelyanskiy, R. Babbush, and H. Neven, Nature Communications 9

2018

Cited alongside, same era.

2021

Later among the works it cites.

Q. Lan, Variational quantum soft actor-critic (2021), arXiv:2112.11921 [quant-ph]

2021

Later among the works it cites.

Y. Liu, S. Arunachalam, and K. Temme, Nature Physics 17

2021

Later among the works it cites.

A. Abbas, D. Sutter, C. Zoufal, A. Lucchi, A. Figalli, and S. Woerner, Nature Computational Science 1

2021

Later among the works it cites.

L. Banchi, J. Pereira, and S. Pirandola, PRX Quantum 2

2021

Later among the works it cites.

K. Bu, D. E. Koh, L. Li, Q. Luo, and Y. Zhang, On the statistical complexity of quantum circuits (2021), arXiv:2101.06154 [quant-ph]

2021

Later among the works it cites.

K. Zhang, M.-H. Hsieh, L. Liu, and D. Tao, arXiv e-prints , arXiv:2112.15002 (2021), arXiv:2112.15002 [quant-ph]

2021

Later among the works it cites.

L. Bittel and M. Kliesch, Phys. Rev. Lett. 127

2021

Later among the works it cites.

Y. Qian, X. Wang, Y. Du, X. Wu, and D. Tao, The dilemma of quantum neural networks (2021), arXiv:2106.04975 [quant-ph]

2021

Later among the works it cites.

V. Saggio, B. E. Asenbeck, A. Hamann, T. Strömberg, P. Schiansky, V. Dunjko, N. Friis, N. C. Harris, M. Hochberg, D. Englund, S. Wölk, H. J. Briegel, and P. Walther, Nature 591

2021

Later among the works it cites.

A. Hamann, V. Dunjko, and S. Wölk, Quantum Machine Intelligence 3

2021

Later among the works it cites.

M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, and P. J. Coles, Nature Reviews Physics 3

2021

Later among the works it cites.

D. Wang, A. Sundaram, R. Kothari, A. Kapoor, and M. Roetteler, in Proceedings of the 38th International Conference on Machine Learning , Proceedings of Machine Learning Research, Vol. 139, edited by M. Meila and T. Zhang (PMLR, 2021) pp. 10916–10926

2021

Later among the works it cites.

C. Ortiz Marrero, M. Kieferová, and N. Wiebe, PRX Quantum 2

2021

Later among the works it cites.

T. Goto, Q. H. Tran, and K. Nakajima, Phys. Rev. Lett. 127

2021

Later among the works it cites.

Y. Kwak, W. J. Yun, S. Jung, J.-K. Kim, and J. Kim, in 2021 International Conference on Information and Communication Technology Convergence (ICTC) (2021) pp. 416–420

2021

Later among the works it cites.

E.-J. Kuo, Y.-L. L. Fang, and S. Y.-C. Chen, Quantum architecture search via deep reinforcement learning (2021), arXiv:2104.07715 [quant-ph]

2021

Later among the works it cites.

E. Ye and S. Y.-C. Chen, Quantum architecture search via continual reinforcement learning (2021), arXiv:2112.05779 [quant-ph]

2021

Later among the works it cites.

K. Zhang, M.-H. Hsieh, L. Liu, and D. Tao, Phys. Rev. Research 3

2021

Later among the works it cites.

Y. Liao, M.-H. Hsieh, and C. Ferrie, Quantum optimization for training quantum neural networks (2021), arXiv:2103.17047 [quant-ph]

2021

Later among the works it cites.

P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs, L. Gilpin, P. Khandelwal, V. Kompella, H. Lin, P. MacAlpine, D. Oller, T. Seno, C. Sherstan, M. D. Thomure, H. Aghabozorgi, L. Barrett, R. Douglas, D. Whitehead, P. Dürr, P. Stone, M. Spranger, and H. Kitano, Nature 602

2022

Closest in time.

J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de las Casas, C. Donner, L. Fritz, C. Galperti, A. Huber, J. Keeling, M. Tsimpoukelli, J. Kay, A. Merle, J.-M. Moret, S. Noury, F. Pesamosca, D. Pfau, O. Sauter, C. Sommariva, S. Coda, B. Duval, A. Fasoli, P. Kohli, K. Kavukcuoglu, D. Hassabis, and M. Riedmiller, Nature 602

2022

Closest in time.