2022

N-Best Hypotheses Reranking for Text-To-SQL Systems

Zeng, Lu, Parthasarathi, Sree Hari Krishnan, Hakkani-Tur, Dilek

Understand

Text-to-SQL task maps natural language utterances to structured queries that can be issued to a database.

State-of-the-art (SOTA) systems rely on finetuning large, pre-trained language models in conjunction with constrained decoding applying a SQL parser.
On the well established Spider dataset, we begin with Oracle studies: specifically, choosing an Oracle hypothesis from a SOTA model's 10-best list, yields a $7.7\%$ absolute improvement in both exact match (EM) and execution (EX) accuracy, showing significant potential improvements with reranking.
Identifying coherence and correctness as reranking approaches, we design a model generating a query plan and propose a heuristic schema linking algorithm.

Built on

“Combining knowledge sources to reorder n-best speech hypothesis lists,”
Manny Rayner, David Carter, Vassilios Digalakis, and Patti Price, · 1994
Earlier work this paper cites.
“Multiple-pass search strategies,”
Richard Schwartz, Long Nguyen, and John Makhoul, · 1996
Earlier work this paper cites.
“Efficient 2-pass n-best decoder.,”
Long Nguyen and Richard M Schwartz, · 1997
Earlier work this paper cites.
“Contextual coherence in natural language processing,”
Robert Porzel and Iryna Gurevych, · 2003
Earlier work this paper cites.
“The rwth phrase-based statistical machine translation system,”
Richard Zens, Oliver Bender, Saša Hasan, Shahram Khadivi, Evgeny Matusov, Jia Xu, Yuqi Zhang, and Hermann Ney, · 2005
Earlier work this paper cites.
“Discriminative reranking for natural language parsing,”
Michael Collins and Terry Koo, · 2005
Earlier work this paper cites.
“Discriminative reranking for semantic parsing,”
Ruifang Ge and Raymond Mooney, · 2006
Earlier work this paper cites.
“Dialogue context-based re-ranking of asr hypotheses,”
Rebecca Jonson, · 2006
Earlier work this paper cites.
“Combining outputs from multiple machine translation systems,”
Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz, and Bonnie Dorr, · 2007
Earlier work this paper cites.
“Goodness: A method for measuring machine translation confidence,”
Nguyen Bach, Fei Huang, and Yaser Al-Onaizan, · 2011
Earlier work this paper cites.
“Word confidence estimation for smt n-best list re-ranking,”
Ngoc-Quang Luong, Laurent Besacier, and Benjamin Lecouteux, · 2014
Earlier work this paper cites.
“A syntactic neural model for general-purpose code generation,”
Original
Pengcheng Yin and Graham Neubig, · 2017
Earlier work this paper cites.
“Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,”
Original
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al., · 2018
Earlier work this paper cites.
“Typesql: Knowledge-based type-aware neural text-to-sql generation,”
Original
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev, · 2018
Earlier work this paper cites.
“Hallucinations in neural machine translation,”
Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo, · 2018
Earlier work this paper cites.

Similar

“Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,”
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer, · 2019
Cited alongside, same era.
“Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases,”
Original
Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al., · 2019
Cited alongside, same era.
“Sparc: cross-domain semantic parsing in context,”
Original
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, et al., · 2019
Cited alongside, same era.
“Towards complex text-to-sql in cross-domain database with intermediate representation,”
Original
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang, · 2019
Cited alongside, same era.
“Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers,”
Original
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson, · 2019
Cited alongside, same era.
“The curious case of neural text degeneration,”
Original
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi, · 2019
Cited alongside, same era.
“Reranking for neural semantic parsing,”
Pengcheng Yin and Graham Neubig, · 2019
Cited alongside, same era.
“Representing schema structure with graph neural networks for text-to-sql parsing,”
Original
Ben Bogin, Matt Gardner, and Jonathan Berant, · 2019
Cited alongside, same era.
“Exploring the limits of transfer learning with a unified text-to-text transformer,”
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, · 2020
Cited alongside, same era.
“Language models are few-shot learners,”
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al., · 2020
Cited alongside, same era.
“Re-examining the role of schema linking in text-to-sql,”
Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua, · 2020
Cited alongside, same era.
“Exploring unexplored generalization challenges for cross-database semantic parsing,”
Alane Laughlin Suhr, Kenton Lee, Ming-Wei Chang, and Pete Shaw, · 2020
Cited alongside, same era.

Then

“Bertrand-dr: Improving text-to-sql using a discriminative re-ranker,”
Original
Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, Chandra Khatri, and Peter Relan, · 2020
Later among the works it cites.
“On faithfulness and factuality in abstractive summarization,”
Original
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald, · 2020
Later among the works it cites.
“Seqgensql–a robust sequence generation model for structured query language,”
Original
Ning Li, Bethany Keller, Mark Butler, and Daniel Cer, · 2020
Later among the works it cites.
“Compositional generalization and natural language variation: Can a semantic parsing approach handle both?,”
Original
Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova, · 2020
Later among the works it cites.
“Evaluating large language models trained on code,”
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al., · 2021
Later among the works it cites.
“Picard: Parsing incrementally for constrained auto-regressive decoding from language models,”
Original
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau, · 2021
Later among the works it cites.
“Webgpt: Browser-assisted question-answering with human feedback,”
Original
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al., · 2021
Later among the works it cites.
“Dynamic hybrid relation exploration network for cross-domain context-dependent semantic parsing,”
Binyuan Hui, Ruiying Geng, Qiyu Ren, Binhua Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, Pengfei Zhu, and Xiaodan Zhu, · 2021
Later among the works it cites.
“Retrieval, re-ranking and multi-task learning for knowledge-base question answering,”
Zhiguo Wang, Patrick Ng, Ramesh Nallapati, and Bing Xiang, · 2021
Later among the works it cites.
“The curious case of hallucinations in neural machine translation,”
Original
Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt, · 2021
Later among the works it cites.
“Language model evaluation beyond perplexity,”
Original
Clara Meister and Ryan Cotterell, · 2021
Later among the works it cites.
“Shadowgnn: Graph projection neural network for text-to-sql parser,”
Original
Zhi Chen, Lu Chen, Yanbin Zhao, Ruisheng Cao, Zihan Xu, Su Zhu, and Kai Yu, · 2021
Later among the works it cites.
“Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases,”
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin, · 2021
Later among the works it cites.
“Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models,”
Original
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al., · 2022
Closest in time.
“Coherence boosting: When your pretrained language model is not paying enough attention,”
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic, · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…