GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment

Understand

The burgeoning size of Large Language Models (LLMs) has led to enhanced capabilities in generating responses, albeit at the expense of increased inference times and elevated resource demands.

• Existing methods of acceleration, predominantly hinged on knowledge distillation, generally necessitate fine-tuning of considerably large models, such as Llama-7B, posing a challenge for average users.
• Furthermore, present techniques for expediting inference and reducing costs operate independently.
• To address these issues, we introduce a novel and intuitive Guidance-based Knowledge Transfer (GKT) framework.