Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective

Understand

Accurate interpretation and visualization of human instructions are crucial for text-to-image (T2I) synthesis.

• However, current models struggle to capture semantic variations from word order changes, and existing evaluations, relying on indirect metrics like text-image similarity, fail to reliably assess these challenges.
• This often obscures poor performance on complex or uncommon linguistic patterns by the focus on frequent word combinations.
• To address these deficiencies, we propose a novel metric called SemVarEffect and a benchmark named SemVarBench, designed to evaluate the causality between semantic variations in inputs and outputs in T2I synthesis.