2023

Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

Li, Jinyang, Hui, Binyuan, Qu, Ge et al.

Understand

Text-to-SQL parsing, which aims at converting natural language instructions into executable SQLs, has gained increasing attention in recent years.

In particular, Codex and ChatGPT have shown impressive results in this task.
However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on database schema with few rows of database contents leaving the gap between academic study and real-world applications.
To mitigate this gap, we present Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks, containing 12,751 pairs of text-to-SQL data and 95 databases with a total size of 33.4 GB, spanning 37 professional domains.

Built on

Space and time savings through large data base compression and dynamic restructuring
Peter Alsberg · 1975
Earlier work this paper cites.
Extensible/rule based query rewrite optimization in starburst
Hamid Pirahesh, Joseph M. Hellerstein, and Waqar Hasan · 1992
Earlier work this paper cites.
Expanding the scope of the ATIS task: The ATIS-3 corpus
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg · 1994
Earlier work this paper cites.
Learning to parse database queries using inductive logic programming
John M. Zelle and Raymond J. Mooney · 1996
Earlier work this paper cites.
Quantitative data cleaning for large databases
Joseph M. Hellerstein · 2008
Earlier work this paper cites.
Sql query optimization methods of relational database system
Dandan Li, Lu Han, and Yi Ding · 2010
Earlier work this paper cites.
Survey on data cleaning
Prerna S. Kulkarni and Jagdish W. Bakal · 2014
Earlier work this paper cites.
Trends in cleaning relational data: Consistency and deduplication
Ihab F. Ilyas and Xu Chu · 2015
Earlier work this paper cites.
A rule based approach for nlp based query processing
Tanzim Mahmud, KM Azharul Hasan, Mahtab Ahmed, and Thwoi Hla Ching Chak · 2015
Earlier work this paper cites.
Revamping sql queries for cost based optimization
Vamsi Krishna Myalapalli and ASN Chakravarthy · 2016
Earlier work this paper cites.
Learning a neural semantic parser from user feedback
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer · 2017
Earlier work this paper cites.
Sqlnet: Generating structured queries from natural language without reinforcement learning
Xiaojun Xu, Chang Liu, and Dawn Song · 2017
Earlier work this paper cites.
Sqlizer: query synthesis from natural language
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig · 2017
Earlier work this paper cites.
Seq2SQL: Generating structured queries from natural language using reinforcement learning
Original
Victor Zhong, Caiming Xiong, and Richard Socher · 2017
Earlier work this paper cites.
An encoder-decoder framework translating natural language to database queries
Ruichu Cai, Boyan Xu, Zhenjie Zhang, Xiaoyan Yang, Zijian Li, and Zhihao Liang · 2018
Earlier work this paper cites.
TypeSQL: Knowledge-based type-aware neural text-to-SQL generation
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev · 2018
Earlier work this paper cites.
Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev · 2018
Earlier work this paper cites.
Towards complex text-to-SQL in cross-domain database with intermediate representation
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang · 2019
Earlier work this paper cites.
Magic mirror in my hand, which is the best in the land? an experimental evaluation of index selection algorithms
Jan Kossmann, Stefan Halfpap, Marcel Jankrift, and Rainer Schlosser · 2020
Earlier work this paper cites.
Batchwise probabilistic incremental data cleaning
Original
Paulo H. Oliveira, Daniel dos Santos Kaster, Caetano Traina, and Ihab F. Ilyas · 2020
Earlier work this paper cites.

Similar

Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020
Cited alongside, same era.
DuSQL: A large-scale and pragmatic Chinese text-to-SQL dataset
Lijie Wang, Ao Zhang, Kun Wu, Ke Sun, Zhenghua Li, Hua Wu, Min Zhang, and Haifeng Wang · 2020
Cited alongside, same era.
Text-to-sql generation for question answering on electronic medical records
Ping Wang, Tian Shi, and Chandan K. Reddy · 2020
Cited alongside, same era.
Grounded adaptation for zero-shot executable semantic parsing
Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer · 2020
Cited alongside, same era.
Database meets artificial intelligence: A survey
Xuanhe Zhou, Chengliang Chai, Guoliang Li, and Ji Sun · 2020
Cited alongside, same era.
LGESQL: Line graph enhanced text-to-SQL model with mixed local and non-local relations
Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu · 2021
Cited alongside, same era.
FinQA: A dataset of numerical reasoning over financial data
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang · 2021
Cited alongside, same era.
Towards robustness of text-to-sql models against synonym substitution
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Pengsheng Huang · 2021
Cited alongside, same era.
Text-to-SQL in the wild: A naturally-occurring dataset based on stack exchange data
Moshe Hazoom, Vibhor Malik, and Ben Bogin · 2021
Cited alongside, same era.
Dynamic hybrid relation exploration network for cross-domain context-dependent semantic parsing
Binyuan Hui, Ruiying Geng, Qiyu Ren, Binhua Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, Pengfei Zhu, and Xiaodan Zhu · 2021
Cited alongside, same era.
KaggleDBQA: Realistic evaluation of text-to-SQL parsers
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson · 2021
Cited alongside, same era.
PICARD: Parsing incrementally for constrained auto-regressive decoding from language models
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau · 2021
Cited alongside, same era.

Then

Large language models are zero-shot reasoners
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa · 2022
Later among the works it cites.
Ehrsql: A practical text-to-sql benchmark for electronic health records
Gyubok Lee, Hyeonji Hwang, Seongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang, Minjoon Seo, Jong-Yeup Kim, and Edward Choi · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Original
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe · 2022
Later among the works it cites.
RASAT: Integrating relational structures into pretrained Seq2Seq model for text-to-SQL
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin · 2022
Later among the works it cites.
A survey on text-to-sql parsing: Concepts, methods, and future directions
Original
Bowen Qin, Binyuan Hui, Lihan Wang, Min Yang, Jinyang Li, Binhua Li, Ruiying Geng, Rongyu Cao, Jian Sun, Luo Si, Fei Huang, and Yongbin Li · 2022
Later among the works it cites.
Evaluating the text-to-sql capabilities of large language models
Original
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau · 2022
Later among the works it cites.
Proton: Probing schema linking information from pre-trained language models for text-to-sql parsing
Lihan Wang, Bowen Qin, Binyuan Hui, Bowen Li, Min Yang, Bailin Wang, Binhua Li, Jian Sun, Fei Huang, Luo Si, and Yongbin Li · 2022
Later among the works it cites.
Wetune: Automatic discovery and verification of query rewrite rules
Zhaoguo Wang, Zhou Zhou, Yicun Yang, Haoran Ding, Gansen Hu, Ding Ding, Chuzhe Tang, Haibo Chen, and Jinyang Li · 2022
Later among the works it cites.
Chain of thought prompting elicits reasoning in large language models
Original
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou · 2022
Later among the works it cites.
UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu · 2022
Later among the works it cites.
Opt: Open pre-trained transformer language models
Original
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer · 2022
Later among the works it cites.
Bridging the generalization gap in text-to-SQL parsing with schema expansion
Chen Zhao, Yu Su, Adam Pauls, and Emmanouil Antonios Platanios · 2022
Later among the works it cites.
Introducing Claude
Anthropic · 2023
Closest in time.
Dr.spider: A diagnostic evaluation benchmark towards text-to-SQL robustness
Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, Steve Ash, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, and Bing Xiang · 2023
Closest in time.
Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing
Original
Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li · 2023
Closest in time.
Augmented language models: a survey
Original
Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom · 2023
Closest in time.
Gpt-4 technical report
Original
OpenAI · 2023
Closest in time.
DIN-SQL: decomposed in-context learning of text-to-sql with self-correction
Original
Mohammadreza Pourreza and Davood Rafiei · 2023
Closest in time.
Research on key performance index prediction of distributed database based on machine learning algorithm
Rong Zhou · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…