Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problems

Understand

Large Language Models (LLMs) have recently achieved impressive performance in math and reasoning benchmarks.

• However, they often struggle with logic problems and puzzles that are relatively easy for humans.
• To further investigate this, we introduce a new benchmark, SearchBench, which contains 11 unique search problems inspired by intuitive puzzles.
• Each SearchBench problem type is equipped with automated pipelines to generate an arbitrary number of instances and analyze the feasibility, correctness, and optimality of LLM-generated solutions.