Fetching the paper…

Reading the bibliography…

2023

A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability

Liu, Aiwei, Hu, Xuming, Wen, Lijie et al.

Original
PDF
Post on X

Understand

This paper presents the first comprehensive analysis of ChatGPT's Text-to-SQL ability.

Given the recent emergence of large-scale conversational language model ChatGPT and its impressive capabilities in both conversational abilities and code generation, we sought to evaluate its Text-to-SQL performance.
We conducted experiments on 12 benchmark datasets with different languages, settings, or scenarios, and the results demonstrate that ChatGPT has strong text-to-SQL abilities.
Although there is still a gap from the current state-of-the-art (SOTA) model performance, considering that the experiment was conducted in a zero-shot scenario, ChatGPT's performance is still impressive.

A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability · Around

Built on

Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901
Earlier work this paper cites.
Sparc: Cross-domain semantic parsing in context
Original
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, et al. 2019b · 1906
Earlier work this paper cites.
A pilot study for chinese sql semantic parsing
Original
Qingkai Min, Yuefeng Shi, and Yue Zhang. 2019 · 1909
Earlier work this paper cites.
Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases
Original
Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. 2019a · 1909
Earlier work this paper cites.
Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers
Original
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2019 · 1911
Earlier work this paper cites.
Structure-grounded pretraining for text-to-sql
Original
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2020 · 2010
Earlier work this paper cites.
Semantic evaluation for text-to-sql with distilled test suites
Original
Ruiqi Zhong, Tao Yu, and Dan Klein. 2020 · 2010
Earlier work this paper cites.
Deep reinforcement learning from human preferences
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017 · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Earlier work this paper cites.

Similar

Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task
Original
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018 · 2018
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020
Cited alongside, same era.
Dusql: A large-scale and pragmatic chinese text-to-sql dataset
Lijie Wang, Ao Zhang, Kun Wu, Ke Sun, Zhenghua Li, Hua Wu, Min Zhang, and Haifeng Wang. 2020 · 2020
Cited alongside, same era.
Lgesql: line graph enhanced text-to-sql model with mixed local and non-local relations
Original
Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021 · 2021
Cited alongside, same era.
Picard: Parsing incrementally for constrained auto-regressive decoding from language models

Then

Towards robustness of text-to-SQL models against natural and realistic adversarial table perturbation
Xinyu Pi, Bing Wang, Yan Gao, Jiaqi Guo, Zhoujun Li, and Jian-Guang Lou. 2022 · 2022
Later among the works it cites.
Rasat: Integrating relational structures into pretrained seq2seq model for text-to-sql
Original
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin. 2022 · 2022
Later among the works it cites.
Evaluating the text-to-sql capabilities of large language models
Original
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022 · 2022
Later among the works it cites.
Proton: Probing schema linking information from pre-trained language models for text-to-sql parsing
Lihan Wang, Bowen Qin, Binyuan Hui, Bowen Li, Min Yang, Bailin Wang, Binhua Li, Jian Sun, Fei Huang, Luo Si, et al. 2022 · 2022
Later among the works it cites.
Cqr-sql: Conversational question reformulation enhanced context-dependent text-to-sql parsers

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021 · 2021

Cited alongside, same era.

Star: Sql guided pre-training for context-dependent text-to-sql parsing

Zefeng Cai, Xiangyu Li, Binyuan Hui, Min Yang, Bowen Li, Binhua Li, Zheng Cao, Weijie Li, Fei Huang, Luo Si, et al. 2022 · 2022

Cited alongside, same era.

Binding language models in symbolic languages

Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. 2022 · 2022

Cited alongside, same era.

Measuring and improving compositional generalization in text-to-sql via component alignment

Yujian Gan, Xinyun Chen, Qiuping Huang, and Matthew Purver. 2022 · 2022

Cited alongside, same era.

Semantic enhanced text-to-sql parsing via iteratively learning schema linking graph

Aiwei Liu, Xuming Hu, Li Lin, and Lijie Wen. 2022 · 2022

Cited alongside, same era.

Towards robustness of text-to-sql models against synonym substitution

Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R Woodward, Jinxia Xie, and Pengsheng Huang. 2021a

Cited in the paper.

Exploring underexplored limitations of cross-domain text-to-sql generalization

Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b

Cited in the paper.

Dongling Xiao, Linzheng Chai, Qian-Wen Zhang, Zhao Yan, Zhoujun Li, and Yunbo Cao. 2022 · 2022

Later among the works it cites.

Mathematical capabilities of chatgpt

Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023 · 2023

Closest in time.

Decoupling the skeleton parsing and schema linking for text-to-sql

Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023 · 2023

Closest in time.

Cross-lingual summarization via chatgpt

Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023 · 2023

Closest in time.

Zero-shot information extraction via chatting with chatgpt

Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023 · 2023

Closest in time.