FedCoT: Federated Chain-of-Thought Distillation for Large Language Models

Understand

Large Language Models (LLMs) have emerged as a transformative force in artificial intelligence, demonstrating exceptional proficiency across various tasks.

• However, their deployment in resource-constrained environments and concerns over user data privacy pose significant challenges.
• In contrast, Small Language Models (SLMs) offer computational efficiency but often lag in performance.
• To address these issues, we propose FedCoT, a federated framework designed for the Chain-of-Thought (CoT) distillation of knowledge from LLMs to SLMs, while ensuring the preservation of clients' data privacy.