Exploring Large Language Models for Relevance Judgments in Tetun

Understand

The Cranfield paradigm has served as a foundational approach for developing test collections, with relevance judgments typically conducted by human assessors.

• However, the emergence of large language models (LLMs) has introduced new possibilities for automating these tasks.
• This paper explores the feasibility of using LLMs to automate relevance assessments, particularly within the context of low-resource languages.
• In our study, LLMs are employed to automate relevance judgment tasks, by providing a series of query-document pairs in Tetun as the input text.