Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing

Understand

The powerful generative abilities of large language models (LLMs) show potential in generating relevance labels for search applications.

• Previous work has found that directly asking about relevancy, such as ``How relevant is document A to query Q?", results in sub-optimal ranking.
• Instead, the pairwise ranking prompting (PRP) approach produces promising ranking performance through asking about pairwise comparisons, e.g., ``Is document A more relevant than document B to query Q?".
• Thus, while LLMs are effective at their ranking ability, this is not reflected in their relevance label generation.