2021

GP: Context-free Grammar Pre-training for Text-to-SQL Parsers

Zhao, Liang, Cao, Hexin, Zhao, Yunsong

Understand

A new method for Text-to-SQL parsing, Grammar Pre-training (GP), is proposed to decode deep relations between question and database.

Firstly, to better utilize the information of databases, a random value is added behind a question word which is recognized as a column, and the new sentence serves as the model input.
Secondly, initialization of vectors for decoder part is optimized, with reference to the former encoding so that question information can be concerned.
Finally, a new approach called flooding level is adopted to get the non-zero training loss which can generalize better results.

Built on

Roberta: A robustly optimized bert pretraining approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · 1907
Earlier work this paper cites.
Dropout: a simple way to prevent neural networks from overfitting
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014) · 1958
Earlier work this paper cites.
Modern natural language interfaces to databases: Composing statistical parsing with semantic tractability
Popescu, A.-M., Armanasu, A., Etzioni, O., Ko, D., & Yates, A. (2004) · 2004
Earlier work this paper cites.
Grappa: Grammar-augmented pre-training for table semantic parsing
Yu, T., Wu, C.-S., Lin, X. V., Wang, B., Chern Tan, Y., Yang, X., Radev, D., Socher, R., & Xiong, C. (2020) · 2009
Earlier work this paper cites.
Template-based question answering over rdf data
Unger, C., Bühmann, L., Lehmann, J., Ngonga Ngomo, A.-C., Gerber, D., & Cimiano, P. (2012) · 2012
Earlier work this paper cites.
Constructing an interactive natural language interface for relational databases
Li, F., & Jagadish, H. (2014) · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Ioffe, S., & Szegedy, C. (2015) · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P., & Ba, J. (2015) · 2015
Earlier work this paper cites.
Pointer networks
Vinyals, O., Fortunato, M., & Jaitly, N. (2015) · 2015
Earlier work this paper cites.

Similar

Deep Learning
Goodfellow, I., Bengio, Y., & Courville, A. (2016) · 2016
Cited alongside, same era.
Rethinking the inception architecture for computer vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016
Cited alongside, same era.
A syntactic neural model for general-purpose code generation
Yin, P., & Neubig, G. (2017) · 2017
Cited alongside, same era.
mixup: Beyond empirical risk minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · 2017
Cited alongside, same era.
Seq2sql: Generating structured queries from natural language using reinforcement learning
Zhong, V., Xiong, C., & Socher, R. (2017) · 2017
Cited alongside, same era.
Deep contextualized word representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · 2018
Cited alongside, same era.
Typesql: Knowledge-based type-aware neural text-to-sql generation
Yu, T., Li, Z., Zhang, Z., Zhang, R., & Radev, D. (2018) · 2018
Cited alongside, same era.
Syntaxsqlnet: Syntax tree networks for complex and cross-domain text-to-sql task
Yu, T., Yasunaga, M., Yang, K., Zhang, R., Wang, D., Li, Z., & Radev, D. (2018a) · 2018
Cited alongside, same era.
Bridging the semantic gap with sql query logs in natural language interfaces to databases
Baik, C., Jagadish, H. V., & Li, Y. (2019) · 2019
Cited alongside, same era.
Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018b)
Cited in the paper.

Then

Global reasoning over database structures for text-to-sql parsing
Bogin, B., Gardner, M., & Berant, J. (2019) · 2019
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · 2019
Later among the works it cites.
Towards complex text-to-sql in cross-domain database with intermediate representation
Guo, J., Zhan, Z., Gao, Y., Xiao, Y., Lou, J.-G., Liu, T., & Zhang, D. (2019) · 2019
Later among the works it cites.
Editing-based sql query generation for cross-domain context-dependent questions
Zhang, R., Yu, T., Er, H. Y., Shim, S., Xue, E., Lin, X. V., Shi, T., Xiong, C., Socher, R., & Radev, D. (2020) · 2019
Later among the works it cites.
Hybridqa: A dataset of multi-hop question answering over tabular and textual data
Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., & Wang, W. Y. (2020) · 2020
Later among the works it cites.
Do we need zero training loss after achieving zero training error?
Ishida, T., Yamane, I., Sakai, T., Niu, G., & Sugiyama, M. (2020) · 2020
Later among the works it cites.
Bridging textual and tabular data for cross-domain text-to-sql semantic parsing
Lin, X. V., Socher, R., & Xiong, C. (2020) · 2020
Later among the works it cites.
Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers
Wang, B., Shin, R., Liu, X., Polozov, O., & Richardson, M. (2020) · 2020
Later among the works it cites.
Tabert: Pretraining for joint understanding of textual and tabular data
Yin, P., Neubig, G., Yih, W.-t., & Riedel, S. (2020) · 2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…