2022

Graph Reinforcement Learning Application to Co-operative Decision-Making in Mixed Autonomy Traffic: Framework, Survey, and Challenges

Liu, Qi, Li, Xueyuan, Li, Zirui et al.

Understand

Proper functioning of connected and automated vehicles (CAVs) is crucial for the safety and efficiency of future intelligent transport systems.

Meanwhile, transitioning to fully autonomous driving requires a long period of mixed autonomy traffic, including both CAVs and human-driven vehicles.
Thus, collaboration decision-making for CAVs is essential to generate appropriate driving behaviors to enhance the safety and efficiency of mixed autonomy traffic.
In recent years, deep reinforcement learning (DRL) has been widely used in solving decision-making problems.

Built on

V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in
1937
Earlier work this paper cites.
R. Bellman, “A markovian decision process,”
1957
Earlier work this paper cites.
R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,”
1992
Earlier work this paper cites.
L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, “Planning and acting in partially observable stochastic domains,”
1998
Earlier work this paper cites.
R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,”
1999
Earlier work this paper cites.
V. Konda and J. Tsitsiklis, “Actor-critic algorithms,”
1999
Earlier work this paper cites.
Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling network architectures for deep reinforcement learning,” in
2003
Earlier work this paper cites.
F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,”
2008
Earlier work this paper cites.
M. E. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey.”
2009
Earlier work this paper cites.
X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,” in
2011
Earlier work this paper cites.
N. De Moura, R. Chatila, K. Evans, S. Chauvier, and E. Dogan, “Ethical decision making for autonomous vehicles,” in
2013
Earlier work this paper cites.
M. Treiber and A. Kesting, “Traffic flow dynamics,”
2013
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,”
Original
2013
Earlier work this paper cites.
J. Erdmann, “Sumo’s lane-changing model,” in
2015
Earlier work this paper cites.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,”
Original
2015
Earlier work this paper cites.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,”
Original
2015
Earlier work this paper cites.
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,”
Original
2016
Earlier work this paper cites.
H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in
2016
Earlier work this paper cites.
S. Gu, T. Lillicrap, I. Sutskever, and S. Levine, “Continuous deep q-learning with model-based acceleration,” in
2016
Earlier work this paper cites.
M. Althoff, M. Koschi, and S. Manzinger, “Commonroad: Composable benchmarks for motion planning on roads,” in
2017
Earlier work this paper cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”
Original
2017
Earlier work this paper cites.
Y. Li, “Deep reinforcement learning: An overview,”
Original
2017
Earlier work this paper cites.
H. Chae, C. M. Kang, B. Kim, J. Kim, C. C. Chung, and J. W. Choi, “Autonomous braking system via deep reinforcement learning,” in
2017
Earlier work this paper cites.
C. Wu, A. Kreidieh, E. Vinitsky, and A. M. Bayen, “Emergent behaviors in mixed-autonomy traffic,” in
2017
Earlier work this paper cites.
M. Fortunato, M. G. Azar, B. Piot, J. Menick, I. Osband, A. Graves, V. Mnih, R. Munos, D. Hassabis, O. Pietquin
Original
2017
Earlier work this paper cites.
M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” in
2017
Earlier work this paper cites.
W. Schwarting, J. Alonso-Mora, and D. Rus, “Planning and decision-making for autonomous vehicles,”
2018
Earlier work this paper cites.
J. Jiang, C. Dun, T. Huang, and Z. Lu, “Graph convolutional reinforcement learning,”
Original
2018
Earlier work this paper cites.
E. Leurent, “An environment for autonomous driving decision-making,”
2018
Earlier work this paper cites.
P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. Wießner, “Microscopic traffic simulation using sumo,” in
2018
Earlier work this paper cites.
E. Vinitsky, A. Kreidieh, L. Le Flem, N. Kheterpal, K. Jang, C. Wu, F. Wu, R. Liaw, E. Liang, and A. M. Bayen, “Benchmarks for reinforcement learning in mixed-autonomy traffic,” in
2018
Earlier work this paper cites.
X. Xu, L. Zuo, X. Li, L. Qian, J. Ren, and Z. Sun, “A reinforcement learning approach to autonomous decision making of intelligent vehicles on highways,”
2018
Earlier work this paper cites.
V. François-Lavet, P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau,
2018
Earlier work this paper cites.
B. Mirchevska, C. Pek, M. Werling, M. Althoff, and J. Boedecker, “High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning,” in
2018
Earlier work this paper cites.
Z. Qiao, K. Muelling, J. Dolan, P. Palanisamy, and P. Mudalige, “Pomdp and hierarchical options mdp with continuous actions for autonomous driving at intersections,” in
2018
Earlier work this paper cites.
M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” in
2018
Earlier work this paper cites.
S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in
2018
Earlier work this paper cites.

Similar

M. Bouton, A. Nakhaei, K. Fujimura, and M. J. Kochenderfer, “Cooperation-aware reinforcement learning for merging in dense traffic,” in
2019
Cited alongside, same era.
C. Yu, X. Wang, X. Xu, M. Zhang, H. Ge, J. Ren, L. Sun, B. Chen, and G. Tan, “Distributed multiagent coordinated learning for autonomous driving in highways based on dynamic coordination graphs,”
2019
Cited alongside, same era.
M. Bouton, A. Nakhaei, K. Fujimura, and M. J. Kochenderfer, “Cooperation-aware reinforcement learning for merging in dense traffic,” in
2019
Cited alongside, same era.
J. Bernhard, S. Pollok, and A. Knoll, “Addressing inherent uncertainty: Risk-sensitive behavior generation for automated driving using distributional reinforcement learning,” in
2019
Cited alongside, same era.
M. Bouton, A. Nakhaei, K. Fujimura, and M. J. Kochenderfer, “Safe reinforcement learning with scene decomposition for navigating complex urban environments,” in
2019
Cited alongside, same era.
A. Alizadeh, M. Moghadam, Y. Bicer, N. K. Ure, U. Yavas, and C. Kurtulus, “Automated lane change decision making using deep reinforcement learning in dynamic and uncertain highway environment,” in
2019
Cited alongside, same era.
D. Liu, M. Brännstrom, A. Backhouse, and L. Svensson, “Learning faster to perform autonomous lane changes by constructing maneuvers from shielded semantic actions,” in
2019
Cited alongside, same era.
F. Pusse and M. Klusch, “Hybrid online pomdp planning and deep reinforcement learning for safer self-driving cars,” in
2019
Cited alongside, same era.
F. Zheng, C. Liu, X. Liu, S. E. Jabari, and L. Lu, “Analyzing the impact of automated vehicles on uncertainty and stability of the mixed traffic flow,”
2020
Cited alongside, same era.
P. Palanisamy, “Multi-agent connected autonomous driving using deep reinforcement learning,” in
2020
Cited alongside, same era.
P. Y. J. Ha, S. Chen, J. Dong, R. Du, Y. Li, and S. Labi, “Leveraging the capabilities of connected and autonomous vehicles and multi-agent reinforcement learning to mitigate highway bottleneck congestion,”
Original
2020
Cited alongside, same era.
N. Naderializadeh, F. H. Hung, S. Soleyman, and D. Khosla, “Graph convolutional value decomposition in multi-agent reinforcement learning,”
Original
2020
Cited alongside, same era.

Then

B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez, “Deep reinforcement learning for autonomous driving: A survey,”
2021
Later among the works it cites.
M. S. Frikha, S. M. Gammar, A. Lahmadi, and L. Andrey, “Reinforcement and deep reinforcement learning for wireless internet of things: A survey,”
2021
Later among the works it cites.
R. N. Boute, J. Gijsbrechts, W. van Jaarsveld, and N. Vanvuchelen, “Deep reinforcement learning for inventory control: A roadmap,”
2021
Later among the works it cites.
L. M. Schmidt, G. Kontes, A. Plinge, and C. Mutschler, “Can you trust your autonomous car? interpretable and verifiably safe reinforcement learning,” in
2021
Later among the works it cites.
S. Kuutti, S. Fallah, and R. Bowden, “Arc: Adversarially robust control policies for autonomous vehicles,” in
2021
Later among the works it cites.
H. Seong, C. Jung, S. Lee, and D. H. Shim, “Learning to drive at unsignalized intersections using attention-based deep reinforcement learning,” in
2021
Later among the works it cites.
S. B. Prathiba, G. Raja, K. Dev, N. Kumar, and M. Guizani, “A hybrid deep reinforcement learning for autonomous vehicles smart-platooning,”
2021
Later among the works it cites.
J. Liu, W. Zhao, and C. Xu, “An efficient on-ramp merging strategy for connected and automated vehicles in multi-lane traffic,”
2021
Later among the works it cites.
G. Wang, J. Hu, Z. Li, and L. Li, “Harmonious lane changing via deep reinforcement learning,”
2021
Later among the works it cites.
D. Kamran, Y. Ren, and M. Lauer, “High-level decisions from a safe maneuver catalog with reinforcement learning for safe and cooperative automated merging,” in
2021
Later among the works it cites.
Z. el abidine Kherroubi, S. Aknine, and R. Bacha, “Novel decision-making strategy for connected and autonomous vehicles in highway on-ramp merging,”
2021
Later among the works it cites.
N. Deshpande, D. Vaufreydaz, and A. Spalanzani, “Navigation in urban environments amongst pedestrians using multi-objective deep reinforcement learning,” in
2021
Later among the works it cites.
G. P. R. Papini, A. Plebe, M. Da Lio, and R. Donà, “A reinforcement learning approach for enacting cautious behaviours in autonomous driving system: Safe speed choice in the interaction with distracted pedestrians,”
2021
Later among the works it cites.
L. Schester and L. E. Ortiz, “Automated driving highway traffic merging using deep multi-agent reinforcement learning in continuous state-action spaces,” in
2021
Later among the works it cites.
T. Wang, Y. Luo, J. Liu, and K. Li, “Multi-objective end-to-end self-driving based on pareto-optimal actor-critic approach,” in
2021
Later among the works it cites.
F. Ye, P. Wang, C.-Y. Chan, and J. Zhang, “Meta reinforcement learning-based lane change strategy for autonomous vehicles,” in
2021
Later among the works it cites.
B. Gangopadhyay, H. Soora, and P. Dasgupta, “Hierarchical program-triggered reinforcement learning agents for automated driving,”
2021
Later among the works it cites.
P. Cai, H. Wang, Y. Sun, and M. Liu, “Dignet: Learning scalable self-driving policies for generic traffic scenarios with graph neural networks,” in
2021
Later among the works it cites.
H. Zhou, A. Aral, I. Brandić, and M. Erol-Kantarci, “Multiagent bayesian deep reinforcement learning for microgrid energy management under communication failures,”
2021
Later among the works it cites.
J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher
Original
2021
Later among the works it cites.
Y. Han, M. Wang, L. Li, C. Roncoli, J. Gao, and P. Liu, “A physics-informed reinforcement learning-based strategy for local and coordinated ramp metering,”
2022
Closest in time.
W. Zhou, D. Chen, J. Yan, Z. Li, H. Yin, and W. Ge, “Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic,”
2022
Closest in time.
Z. Bai, P. Hao, W. Shangguan, B. Cai, and M. J. Barth, “Hybrid reinforcement learning-based eco-driving strategy for connected and automated vehicles at signalized intersections,”
2022
Closest in time.
J. Hu, X. Li, Y. Cen, Q. Xu, X. Zhu, and W. Hu, “A roadside decision-making methodology based on deep reinforcement learning to simultaneously improve the safety and efficiency of merging zone,”
2022
Closest in time.
R. Trumpp, H. Bayerlein, and D. Gesbert, “Modeling interactions of autonomous vehicles and pedestrians with deep multi-agent reinforcement learning for collision avoidance,” in
2022
Closest in time.
F. Yang, X. Li, Q. Liu, Z. Li, and X. Gao, “Generalized single-vehicle-based graph reinforcement learning for decision-making in autonomous driving,”
2022
Closest in time.
X. Gao, X. Li, Q. Liu, Z. Li, F. Yang, and T. Luan, “Multi-agent decision-making modes in uncertain interactive traffic scenarios via graph convolution-based deep reinforcement learning,”
2022
Closest in time.
M. Klimke, B. Völz, and M. Buchholz, “Cooperative behavioral planning for automated driving using graph neural networks,”
Original
2022
Closest in time.
——, “Dq-gat: Towards safe and efficient autonomous driving with deep q-learning and graph attention networks,”
2022
Closest in time.
J. Liu, Z. Wang, and L. Zhang, “Integrated vehicle-following control for four-wheel-independent-drive electric vehicles against non-ideal v2x communication,”
2022
Closest in time.
R. Wang, Y. Zhang, G. Fortino, Q. Guan, J. Liu, and J. Song, “Software escalation prediction based on deep learning in the cognitive internet of vehicles,”
2022
Closest in time.
J. Blumenkamp, S. Morad, J. Gielis, Q. Li, and A. Prorok, “A framework for real-world multi-robot systems running decentralized gnn-based policies,” in
2022
Closest in time.
X. He, J. Stapel, M. Wang, and R. Happee, “Modelling perceived risk and trust in driving automation reacting to merging and braking vehicles,”
2022
Closest in time.
M. Liu, J. Zhao, S. Hoogendoorn, and M. Wang, “A single-layer approach for joint optimization of traffic signals and cooperative vehicle trajectories at isolated intersections,”
2022
Closest in time.
N. Lichtlé, E. Vinitsky, G. Gunter, A. Velu, and A. M. Bayen, “Fuel consumption reduction of multi-lane road networks using decentralized mixed-autonomy control,” in
2073
Closest in time.
Z. Yan and C. Wu, “Reinforcement learning for mixed autonomy intersections,” in
2094
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…