Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding

Understand

Automated audio captioning (AAC) is an audio-to-text task to describe audio contents in natural language.

• Recently, the advancements in large language models (LLMs), with improvements in training approaches for audio encoders, have opened up possibilities for improving AAC.
• Thus, we explore enhancing AAC from three aspects: 1) a pre-trained audio encoder via consistent ensemble distillation (CED) is used to improve the effectivity of acoustic tokens, with a querying transformer (Q-Former) bridging the modality gap to LLM and compress acoustic tokens; 2) we investigate the advantages of using a Llama 2 with 7B parameters as the decoder; 3) another pre-trained LLM corrects text errors caused by insufficient training data and annotation ambiguities.
• Both the audio encoder and text decoder are optimized by low-rank adaptation (LoRA).