Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models

Understand

Large Language Models (LLMs) have shown strong potential for tabular data generation by modeling textualized feature-value pairs.

• However, tabular data inherently exhibits sparse feature-level dependencies, where many feature interactions are structurally insignificant.
• This creates a fundamental mismatch as LLMs' self-attention mechanism inevitably distributes focus across all pairs, diluting attention on critical relationships, particularly in datasets with complex dependencies or semantically ambiguous features.
• To address this limitation, we propose GraDe (Graph-Guided Dependency Learning), a novel method that explicitly integrates sparse dependency graphs into LLMs' attention mechanism.