Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation

Understand

We explore the use of long-context capabilities in large language models to create synthetic reading comprehension data from entire books.

• Previous efforts to construct such datasets relied on crowd-sourcing, but the emergence of transformers with a context size of 1 million or more tokens now enables entirely automatic approaches.
• Our objective is to test the capabilities of LLMs to analyze, understand, and reason over problems that require a detailed comprehension of long spans of text, such as questions involving character arcs, broader themes, or the consequences of early actions later in the story.
• We propose a holistic pipeline for automatic data generation including question generation, answering, and model scoring using an ``Evaluator''.