Measuring agreement among several raters classifying subjects into one or more (hierarchical) categories: A generalization of Fleiss' kappa

Understand

Cohen's and Fleiss' kappa are well-known measures of inter-rater agreement, but they restrict each rater to selecting only one category per subject.

• This limitation is consequential in contexts where subjects may belong to multiple categories, such as psychiatric diagnoses involving multiple disorders or classifying interview snippets into multiple codes of a codebook.
• We propose a generalized version of Fleiss' kappa, which accommodates multiple raters assigning subjects to one or more nominal categories.
• Our proposed $\kappa$ statistic can incorporate category weights based on their importance and account for hierarchical category structures, such as primary disorders with sub-disorders.