Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test

Understand

The Shuffle Test is the most common task to evaluate whether NLP models can measure coherence in text.

• Most recent work uses direct supervision on the task; we show that by simply finetuning a RoBERTa model, we can achieve a near perfect accuracy of 97.8%, a state-of-the-art.
• We argue that this outstanding performance is unlikely to lead to a good model of text coherence, and suggest that the Shuffle Test should be approached in a Zero-Shot setting: models should be evaluated without being trained on the task itself.
• We evaluate common models in this setting, such as Generative and Bi-directional Transformers, and find that larger architectures achieve high-performance out-of-the-box.