DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation

Understand

Large language models (LLMs) have achieved significant success across various domains.

• However, training these LLMs typically involves substantial memory and computational costs during both forward and backward propagation.
• While parameter-efficient fine-tuning (PEFT) considerably reduces the training memory associated with parameters, it does not address the significant computational costs and activation memory.
• In this paper, we propose Dropping Backward Propagation (DropBP), a novel approach designed to reduce computational costs and activation memory while maintaining accuracy.