Automatic Construction of Evaluation Suites for Natural Language Generation Datasets

Understand

Machine learning approaches applied to NLP are often evaluated by summarizing their performance in a single number, for example accuracy.

• Since most test sets are constructed as an i.i.d.
• sample from the overall data, this approach overly simplifies the complexity of language and encourages overfitting to the head of the data distribution.
• As such, rare language phenomena or text about underrepresented groups are not equally included in the evaluation.