Hyperbolic Fine-Tuning for Large Language Models

Understand

Large language models (LLMs) have demonstrated remarkable performance across various tasks.

• However, it remains an open question whether the default Euclidean space is the most suitable choice for LLMs.
• In this study, we investigate the geometric characteristics of LLMs, focusing specifically on tokens and their embeddings.
• Our findings reveal that token frequency follows a power-law distribution, where high-frequency tokens (e.g., the, that ) constitute the minority, while low-frequency tokens (e.g., apple, dog) constitute the majority.