Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning

Understand

While fine-tuning large language models (LLMs) for specific tasks often yields impressive results, it comes at the cost of memory inefficiency due to back-propagation in gradient-based training.

• Memory-efficient Zeroth-order (MeZO) optimizers, recently proposed to address this issue, only require forward passes during training, making them more memory-friendly.
• However, compared with exact gradients, ZO-based gradients usually exhibit an estimation error, which can significantly hurt the optimization process, leading to slower convergence and suboptimal solutions.
• In addition, we find that the estimation error will hurt more when adding to large weights instead of small weights.