Find Everything: A General Vision Language Model Approach to Multi-Object Search

Understand

The Multi-Object Search (MOS) problem involves navigating to a sequence of locations to maximize the likelihood of finding target objects while minimizing travel costs.

• In this paper, we introduce a novel approach to the MOS problem, called Finder, which leverages vision language models (VLMs) to locate multiple objects across diverse environments.
• Specifically, our approach introduces multi-channel score maps to track and reason about multiple objects simultaneously during navigation, along with a score map technique that combines scene-level and object-level semantic correlations.
• Experiments in both simulated and real-world settings showed that Finder outperforms existing methods using deep reinforcement learning and VLMs.