MIMICS-Duo: Offline & Online Evaluation of Search Clarification

Understand

Asking clarification questions is an active area of research; however, resources for training and evaluating search clarification methods are not sufficient.

• To address this issue, we describe MIMICS-Duo, a new freely available dataset of 306 search queries with multiple clarifications (a total of 1,034 query-clarification pairs).
• MIMICS-Duo contains fine-grained annotations on clarification questions and their candidate answers and enhances the existing MIMICS datasets by enabling multi-dimensional evaluation of search clarification methods, including online and offline evaluation.
• We conduct extensive analysis to demonstrate the relationship between offline and online search clarification datasets and outline several research directions enabled by MIMICS-Duo.