When Life gives you LLMs, make LLM-ADE: Large Language Models with Adaptive Data Engineering

Understand

This paper presents the LLM-ADE framework, a novel methodology for continued pre-training of large language models (LLMs) that addresses the challenges of catastrophic forgetting and double descent.

• LLM-ADE employs dynamic architectural adjustments, including selective block freezing and expansion, tailored to specific datasets.
• This strategy enhances model adaptability to new data while preserving previously acquired knowledge.
• We demonstrate LLM-ADE's effectiveness on the TinyLlama model across various general knowledge benchmarks, showing significant performance improvements without the drawbacks of traditional continuous training methods.