AVA: an Automatic eValuation Approach to Question Answering Systems

Understand

We introduce AVA, an automatic evaluation approach for Question Answering, which given a set of questions associated with Gold Standard answers, can estimate system Accuracy.

• AVA uses Transformer-based language models to encode question, answer, and reference text.
• This allows for effectively measuring the similarity between the reference and an automatic answer, biased towards the question semantics.
• To design, train and test AVA, we built multiple large training, development, and test sets on both public and industrial benchmarks.