ANALOGICAL -- A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models

Understand

Over the past decade, analogies, in the form of word-level analogies, have played a significant role as an intrinsic measure of evaluating the quality of word embedding methods such as word2vec.

• Modern large language models (LLMs), however, are primarily evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE, and there are only a few investigations on whether LLMs can draw analogies between long texts.
• In this paper, we present ANALOGICAL, a new benchmark to intrinsically evaluate LLMs across a taxonomy of analogies of long text with six levels of complexity -- (i) word, (ii) word vs.
• sentence, (iii) syntactic, (iv) negation, (v) entailment, and (vi) metaphor.