ROMA: a Read-Only-Memory-based Accelerator for QLoRA-based On-Device LLM

Understand

As large language models (LLMs) demonstrate powerful capabilities, deploying them on edge devices has become increasingly crucial, offering advantages in privacy and real-time interaction.

• QLoRA has emerged as the standard approach for on-device LLMs, leveraging quantized models to reduce memory and computational costs while utilizing LoRA for task-specific adaptability.
• In this work, we propose ROMA, a QLoRA accelerator with a hybrid storage architecture that uses ROM for quantized base models and SRAM for LoRA weights and KV cache.
• Our insight is that the quantized base model is stable and converged, making it well-suited for ROM storage.