The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

Understand

While large language models (LLMs) are still being adopted to new domains and utilized in novel applications, we are experiencing an influx of the new generation of foundation models, namely multi-modal large language models (MLLMs).

• These models integrate verbal and visual information, opening new possibilities to demonstrate more complex reasoning abilities at the intersection of the two modalities.
• However, despite the revolutionizing prospect of MLLMs, our understanding of their reasoning abilities is limited.
• In this study, we assess the nonverbal abstract reasoning abilities of open-source and closed-source MLLMs using variations of Raven's Progressive Matrices.