Semantic Evaluation for Text-to-SQL with Distilled Test Suites

Understand

We propose test suite accuracy to approximate semantic accuracy for Text-to-SQL models.

• Our method distills a small test suite of databases that achieves high code coverage for the gold query from a large number of randomly generated databases.
• At evaluation time, it computes the denotation accuracy of the predicted queries on the distilled test suite, hence calculating a tight upper-bound for semantic accuracy efficiently.
• We use our proposed method to evaluate 21 models submitted to the Spider leader board and manually verify that our method is always correct on 100 examples.