Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark

Understand

Text-to-SQL, which involves translating natural language into Structured Query Language (SQL), is crucial for enabling broad access to structured databases without expert knowledge.

• However, designing models for such tasks is challenging due to numerous factors, including the presence of 'noise,' such as ambiguous questions and syntactical errors.
• This study provides an in-depth analysis of the distribution and types of noise in the widely used BIRD-Bench benchmark and the impact of noise on models.
• While BIRD-Bench was created to model dirty and noisy database values, it was not created to contain noise and errors in the questions and gold queries.