Fetching the paper…

Reading the bibliography…

2024

Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code

Galimzyanov, Timur, Titov, Sergey, Golubev, Yaroslav et al.

Original
PDF
Post on X

Understand

This paper introduces the human-curated PandasPlotBench dataset, designed to evaluate language models' effectiveness as assistants in visual data exploration.

Our benchmark focuses on generating code for visualizing tabular data - such as a Pandas DataFrame - based on natural language instructions, complementing current evaluation tools and expanding their scope.
The dataset includes 175 unique tasks.
Our experiments assess several leading Large Language Models (LLMs) across three visualization libraries: Matplotlib, Seaborn, and Plotly.

Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code · Around

Built on

J. D. Hunter, “Matplotlib: A 2D graphics environment,” Computing in science & engineering , vol. 9, no. 03, pp. 90–95, 2007
2007
Earlier work this paper cites.
The pandas development team, “pandas-dev/pandas: Pandas,” 2020. [Online]. Available: https://doi.org/10.5281/zenodo.3509134
2020
Earlier work this paper cites.
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al. , “CodeBERT: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020
Original
2020
Earlier work this paper cites.
Y. Luo, J. Tang, and G. Li, “nvBench: A large-scale synthesized dataset for cross-domain natural language to visualization task,” arXiv preprint arXiv:2112.12926 , 2021
Original
2021
Earlier work this paper cites.
M. L. Waskom, “Seaborn: statistical data visualization,” Journal of Open Source Software , vol. 6, no. 60, p. 3021, 2021
2021
Earlier work this paper cites.
P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in CHI conference on human factors in computing systems extended abstracts , 2022, pp. 1–7
2022
Earlier work this paper cites.
M. Nejjar, L. Zacharias, F. Stiehle, and I. Weber, “LLMs for science: Usage for code generation and data analysis,” Journal of Software: Evolution and Process , p. e2723, 2023
2023
Earlier work this paper cites.
Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.-t. Yih, D. Fried, S. Wang, and T. Yu, “DS-1000: A natural and reliable benchmark for data science code generation,” in International Conference on Machine Learning . PMLR, 2023, pp. 18 319–18 345
2023
Earlier work this paper cites.

Similar

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023
Original
2023
Cited alongside, same era.
C. Beasley and A. Abouzied, “Pipe(line) dreams: Fully automated end-to-end analysis and visualization,” in Proceedings of the 2024 Workshop on Human-In-the-Loop Data Analytics , 2024, pp. 1–7
2024
Cited alongside, same era.
Z. Yang, Z. Zhou, S. Wang, X. Cong, X. Han, Y. Yan, Z. Liu, Z. Tan, P. Liu, D. Yu et al. , “MatPlotAgent: Method and evaluation for LLM-based agentic scientific data visualization,” arXiv preprint arXiv:2402.11453 , 2024
Original
2024
Cited alongside, same era.
C. Wu, Y. Ge, Q. Guo, J. Wang, Z. Liang, Z. Lu, Y. Shan, and P. Luo, “Plot2Code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots,” arXiv preprint arXiv:2405.07990 , 2024
Original
2024
Cited alongside, same era.

Then

OpenAI, “Hello GPT-4o,” 2024, accessed: 2025-02-10. [Online]. Available: https://openai.com/index/hello-gpt-4o/
2025
Closest in time.
Anthropic, “The Claude 3 model family: Opus, Sonnet, Haiku,” 2024, accessed: 2025-02-10. [Online]. Available: https://www.anthropic.com
2025
Closest in time.
Plotly Technologies Inc., “Collaborative data science,” Montreal, QC, 2015, accessed: 2025-02-10. [Online]. Available: https://plot.ly
2025
Closest in time.
T. Galimzyanov, S. Titov, Y. Golubev, and E. Bogomolov, “HuggingFace Page with PandasPlotBench,” 2024, accessed: 2025-02-10. [Online]. Available: https://huggingface.co/datasets/JetBrains-Research/PandasPlotBench
2025
Closest in time.
——, “Code for running the benchmark,” 2024, accessed: 2025-02-10. [Online]. Available: https://github.com/JetBrains-Research/PandasPlotBench
2025

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

C. Shi, C. Yang, Y. Liu, B. Shui, J. Wang, M. Jing, L. Xu, X. Zhu, S. Li, Y. Zhang et al. , “ChartMimic: Evaluating LMM’s cross-modal reasoning capability via chart-to-code generation,” arXiv preprint arXiv:2406.09961 , 2024

2024

Cited alongside, same era.

G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al. , “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530 , 2024

2024

Cited alongside, same era.

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

Original

2024

Cited alongside, same era.

J. A. Jansen, A. Manukyan, N. Al Khoury, and A. Akalin, “Leveraging large language models for data analysis automation,” PloS one , vol. 20, no. 2, p. e0317084, 2025

2025

Cited alongside, same era.

Closest in time.

——, “Supplementary materials,” 2024, accessed: 2025-02-10. [Online]. Available: https://github.com/JetBrains-Research/PandasPlotBench/tree/main/paper_supp_info

2025

Closest in time.

“MatPlotLib Gallery,” accessed: 2025-02-10. [Online]. Available: https://matplotlib.org/stable/gallery/index.html

2025

Closest in time.

OpenAI, “GPT-4V(ision) System Card,” 2023, accessed: 2025-02-10. [Online]. Available: https://cdn.openai.com/papers/GPTV_System_Card.pdf

2025

Closest in time.