By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

Understand

Large language models (LLMs) have demonstrated exceptional abilities across various domains.

• However, utilizing LLMs for ubiquitous sensing applications remains challenging as existing text-prompt methods show significant performance degradation when handling long sensor data sequences.
• We propose a visual prompting approach for sensor data using multimodal LLMs (MLLMs).
• We design a visual prompt that directs MLLMs to utilize visualized sensor data alongside the target sensory task descriptions.