2024

Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translation

Ranaldi, Federico, Ruzzetti, Elena Sofia, Onorati, Dario et al.

Understand

Understanding textual description to generate code seems to be an achieved capability of instruction-following Large Language Models (LLMs) in zero-shot scenario.

However, there is a severe possibility that this translation ability may be influenced by having seen target textual descriptions and the related code.
This effect is known as Data Contamination.
In this study, we investigate the impact of Data Contamination on the performance of GPT-3.5 in the Text-to-SQL code-generating tasks.

Built on

An efficient easily adaptable system for interpreting natural language queries
David H.D. Warren and Fernando C.N. Pereira. 1982 · 1982
Earlier work this paper cites.
Translating questions to SQL queries with generative parsers discriminatively reranked
Alessandra Giordani and Alessandro Moschitti. 2012 · 2012
Earlier work this paper cites.
Neural enquirer: Learning to query tables in natural language
Pengcheng Yin, Zhengdong Lu, Hang Li, and Kao Ben. 2016 · 2016
Earlier work this paper cites.
Cosette: An automated prover for sql
Shumo Chu, Chenglong Wang, Konstantin Weitz, and Alvin Cheung. 2017 · 2017
Earlier work this paper cites.
Sqlnet: Generating structured queries from natural language without reinforcement learning
Original
Xiaojun Xu, Chang Liu, and Dawn Song. 2017 · 2017
Earlier work this paper cites.
Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018 · 2018
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task
Original
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019 · 2019
Earlier work this paper cites.

Similar

Semantic evaluation for text-to-SQL with distilled test suites
Original
Ruiqi Zhong, Tao Yu, and Dan Klein. 2020 · 2020
Cited alongside, same era.
Extracting training data from large language models
Original
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021 · 2021
Cited alongside, same era.
Data contamination: From memorization to exploitation
Original
Inbal Magar and Roy Schwartz. 2022 · 2022
Cited alongside, same era.
Evaluating the text-to-sql capabilities of large language models
Original
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022 · 2022
Cited alongside, same era.
Speak, memory: An archaeology of books known to chatgpt/gpt-4
Original
Kent K. Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023 · 2023
Cited alongside, same era.
Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks
Original
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023 · 2023
Cited alongside, same era.
Investigating data contamination in modern benchmarks for large language models
Original
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2023 · 2023
Cited alongside, same era.
Text-to-sql empowered by large language models: A benchmark evaluation
Original
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023 · 2023
Cited alongside, same era.
Gpt-3.5turbo
OpenAI. 2023a
Cited in the paper.
Gpt’s family
OpenAI. 2023b
Cited in the paper.
Act-sql: In-context learning for text-to-sql with automatically-generated chain-of-thought
Original
Hanchong Zhang, Ruisheng Cao, Lu Chen, Hongshen Xu, and Kai Yu. 2023a
Cited in the paper.
Self-edit: Fault-aware code editor for code generation
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023b
Cited in the paper.

Then

Time travel in llms: Tracing data contamination in large language models
Original
Shahriar Golchin and Mihai Surdeanu. 2023 · 2023
Later among the works it cites.
Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls
Original
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023 · 2023
Later among the works it cites.
Din-sql: Decomposed in-context learning of text-to-sql with self-correction
Original
Mohammadreza Pourreza and Davood Rafiei. 2023 · 2023
Later among the works it cites.
PreCog: Exploring the relation between memorization and performance in pre-trained language models
Leonardo Ranaldi, Elena Sofia Ruzzetti, and Fabio Massimo Zanzotto. 2023 · 2023
Later among the works it cites.
Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark
Original
Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023 · 2023
Later among the works it cites.
Codet5+: Open code large language models for code understanding and generation
Original
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023 · 2023
Later among the works it cites.
Evaluating instruction-tuned large language models on code comprehension and generation
Original
Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou. 2023 · 2023
Later among the works it cites.
Investigating data contamination for pre-training language models
Original
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. 2024 · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…