LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Driving

A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in Conference on Robot Learning . PMLR, 2017, pp. 1–16

2017

Earlier work this paper cites.

E. Leurent, “An environment for autonomous driving decision-making,” https://github.com/eleurent/highway-env, 2018

2018

Earlier work this paper cites.

V. Ramanishka, Y.-T. Chen, T. Misu, and K. Saenko, “Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 7699–7707

2018

Earlier work this paper cites.

J. Kim, T. Misu, Y.-T. Chen, A. Tawari, and J. Canny, “Grounding human-to-vehicle advice for self-driving vehicles,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 10 591–10 599

2019

Earlier work this paper cites.

H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuScenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 11 621–11 631

2020

Earlier work this paper cites.

I. Bae, J. Moon, J. Jhung, H. Suk, T. Kim, H. Park, J. Cha, J. Kim, D. Kim, and S. Kim, “Self-driving like a human driver instead of a robocar: Personalized comfortable driving experience for autonomous vehicles,” arXiv preprint arXiv:2001.03908 , 2020

Original

2020

Earlier work this paper cites.

H. Caesar, J. Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and S. Omari, “NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles,” arXiv preprint arXiv:2106.11810 , 2021

Original

2021

Earlier work this paper cites.

B. Yu, C. Chen, J. Tang, S. Liu, and J.-L. Gaudiot, “Autonomous vehicles digital twin: A practical paradigm for autonomous driving system development,” Computer , vol. 55, no. 9, pp. 26–34, 2022

2022

Earlier work this paper cites.

OpenAI, “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

Original

2023

Earlier work this paper cites.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

Original

2023

Earlier work this paper cites.

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al. , “PaLM: Scaling language modeling with pathways,” Journal of Machine Learning Research , vol. 24, no. 240, pp. 1–113, 2023

2023

Earlier work this paper cites.

OpenAI, “GPT-4V(ision) system card,” https://openai.com/research/gpt-4v-system-card, 2023

2023

Earlier work this paper cites.

X. Li, Y. Bai, P. Cai, L. Wen, D. Fu, B. Zhang, X. Yang, X. Cai, T. Ma, J. Guo, X. Gao, M. Dou, B. Shi, Y. Liu, L. He, and Y. Qiao, “Towards knowledge-driven autonomous driving,” arXiv preprint arXiv:2312.04316 , 2023

Original

2023

Earlier work this paper cites.

Y. Huang, Y. Chen, and Z. Li, “Applications of large scale foundation models for autonomous driving,” arXiv preprint arXiv:2311.12144 , 2023

Original

2023

Cited alongside, same era.

L. Wen, D. Fu, X. Li, X. Cai, T. Ma, P. Cai, M. Dou, B. Shi, L. He, and Y. Qiao, “Dilu: A knowledge-driven approach to autonomous driving with large language models,” arXiv preprint arXiv:2309.16292 , 2023

Original

2023

Cited alongside, same era.

S. Sharan, F. Pittaluga, M. Chandraker et al. , “Llm-assist: Enhancing closed-loop planning with language-based reasoning,” arXiv preprint arXiv:2401.00125 , 2023

Original

2023

Cited alongside, same era.

W. Wang, J. Xie, C. Hu, H. Zou, J. Fan, W. Tong, Y. Wen, S. Wu, H. Deng, Z. Li et al. , “DriveMLM: Aligning multi-modal large language models with behavioral planning states for autonomous driving,” arXiv preprint arXiv:2312.09245 , 2023

Original

2023

Cited alongside, same era.

Y. Ma, Y. Cao, J. Sun, M. Pavone, and C. Xiao, “Dolphins: Multimodal language model for driving,” arXiv preprint arXiv:2312.00438 , 2023

Original

2023

Later among the works it cites.

D. Wu, W. Han, T. Wang, Y. Liu, X. Zhang, and J. Shen, “Language prompt for autonomous driving,” arXiv preprint arXiv:2309.04379 , 2023

Original

2023

Later among the works it cites.

L. Chen, O. Sinavski, J. Hünermann, A. Karnsund, A. J. Willmott, D. Birch, D. Maund, and J. Shotton, “Driving with LLMs: Fusing object-level vector modality for explainable autonomous driving,” arXiv preprint arXiv:2310.01957 , 2023

Original

2023

Later among the works it cites.

T. Qian, J. Chen, L. Zhuo, Y. Jiao, and Y.-G. Jiang, “Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario,” arXiv preprint arXiv:2305.14836 , 2023

Original

2023

Later among the works it cites.