Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models

Understand

The ability to quickly learn a new task with minimal instruction - known as few-shot learning - is a central aspect of intelligent agents.

• Classical few-shot benchmarks make use of few-shot samples from a single modality, but such samples may not be sufficient to characterize an entire concept class.
• In contrast, humans use cross-modal information to learn new concepts efficiently.
• In this work, we demonstrate that one can indeed build a better ${\bf visual}$ dog classifier by ${\bf read}$ing about dogs and ${\bf listen}$ing to them bark.