Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

Understand

When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training.

• It is often conjectured that this can teach the model the behavior of hallucinating factually incorrect responses, as the model is trained to generate facts that are not grounded in its pre-existing knowledge.
• In this work, we study the impact of such exposure to new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge.
• To this end, we design a controlled setup, focused on closed-book QA, where we vary the proportion of the fine-tuning examples that introduce new knowledge.