Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection

Understand

Data selection in instruction tuning emerges as a pivotal process for acquiring high-quality data and training instruction-following large language models (LLMs), but it is still a new and unexplored research area for vision-language models (VLMs).

• Existing data selection approaches on LLMs either rely on single unreliable scores, or use downstream tasks for selection, which is time-consuming and can lead to potential over-fitting on the chosen evaluation datasets.
• To address this challenge, we introduce a novel dataset selection method, Self-Filter, that utilizes the VLM itself as a filter.
• This approach is inspired by the observation that VLMs benefit from training with the most challenging instructions.