Generative AI as a metacognitive agent: A comparative mixed-method study with human participants on ICF-mimicking exam performance

Understand

This study investigates the metacognitive capabilities of Large Language Models relative to human metacognition in the context of the International Coaching Federation ICF mimicking exam, a situational judgment test related to coaching competencies.

• Using a mixed method approach, we assessed the metacognitive performance, including sensitivity, accuracy in probabilistic predictions, and bias, of human participants and five advanced LLMs (GPT-4, Claude-3-Opus 3, Mistral Large, Llama 3, and Gemini 1.5 Pro).
• The results indicate that LLMs outperformed humans across all metacognitive metrics, particularly in terms of reduced overconfidence, compared to humans.
• However, both LLMs and humans showed less adaptability in ambiguous scenarios, adhering closely to predefined decision frameworks.