Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation

Understand

Recently, Large Language Models (LLMs) have demonstrated great potential in natural language-driven molecule discovery.

• However, existing datasets and benchmarks for molecule-text alignment are predominantly built on one-to-one mappings, measuring LLMs' ability to retrieve a single, pre-defined answer, rather than their creative potential to generate diverse, yet equally valid, molecular candidates.
• To address this critical gap, we propose Speak-to-Structure (S^2-Bench), the first benchmark to evaluate LLMs in open-domain natural language-driven molecule generation.
• S^2-Bench is specifically designed for one-to-many relationships, challenging LLMs to exhibit genuine molecular understanding and open-ended generation capabilities.