HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understanding

Understand

Comprehending extended audiovisual experiences remains challenging for computational systems, particularly temporal integration and cross-modal associations fundamental to human episodic memory.

• We introduce HippoMM, a computational cognitive architecture that maps hippocampal mechanisms to solve these challenges.
• Rather than relying on scaling or architectural sophistication, HippoMM implements three integrated components: (i) Episodic Segmentation detects audiovisual input changes to split videos into discrete episodes, mirroring dentate gyrus pattern separation; (ii) Memory Consolidation compresses episodes into summaries with key features preserved, analogous to hippocampal memory formation; and (iii) Hierarchical Memory Retrieval first searches semantic summaries, then escalates via temporal window expansion around seed segments for cross-modal queries, mimicking CA3 pattern completion.
• These components jointly create an integrated system exceeding the sum of its parts.