Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilities

Verification of Forecasts Expressed in Terms of Probability

Glenn W. Brier. 1950 · 1950

Earlier work this paper cites.

Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods

John Platt. 2000 · 2000

Earlier work this paper cites.

Transforming classifier scores into accurate multiclass probability estimates

Bianca Zadrozny and Charles Elkan. 2002 · 2002

Earlier work this paper cites.

Predicting good probabilities with supervised learning

Alexandru Niculescu-Mizil and Rich Caruana. 2005 · 2005

Earlier work this paper cites.

Selective classification for deep neural networks

Original

Yonatan Geifman and Ran El-Yaniv. 2017 · 2017

Earlier work this paper cites.

On calibration of modern neural networks

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017a · 2017

Earlier work this paper cites.

Calibration of encoder decoder models for neural machine translation

Original

Aviral Kumar and Sunita Sarawagi. 2019 · 2019

Earlier work this paper cites.

Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task

Original

Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019 · 2019

Earlier work this paper cites.

Calibration of pre-trained transformers

Shrey Desai and Greg Durrett. 2020 · 2020

Earlier work this paper cites.

Language models (mostly) know what they know

Original

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022 · 2022

Earlier work this paper cites.

Uncertainty in natural language generation: From theory to applications

Original

Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, R. Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023 · 2023

Earlier work this paper cites.

Benchmarking and improving text-to-SQL generation under ambiguity

Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023 · 2023

Cited alongside, same era.

Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023 · 2023

Cited alongside, same era.

Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls

Original

Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023 · 2023

Cited alongside, same era.

Tce: A test-based approach to measuring calibration error

Original

Takuo Matsubara, Niek Tax, Richard Mudd, and Ido Guy. 2023 · 2023

Cited alongside, same era.

Self-evaluation improves selective generation in large language models

Large language models must be taught to know what they don’t know

Original

Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. 2024 · 2024

Closest in time.

Codes: Towards building open-source language models for text-to-sql

Original

Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024 · 2024

Closest in time.

Code llama: Open foundation models for code

Original

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024 · 2024

Closest in time.

Zero and few-shot semantic parsing with ambiguous inputs

Elias Stengel-Eskin, Kyle Rawlins, and Benjamin Van Durme. 2024 · 2024

Closest in time.