Robust Deep Multi-modal Learning Based on Gated Information Fusion Network

Understand

The goal of multi-modal learning is to use complimentary information on the relevant task provided by the multiple modalities to achieve reliable and robust performance.

• Recently, deep learning has led significant improvement in multi-modal learning by allowing for the information fusion in the intermediate feature levels.
• This paper addresses a problem of designing robust deep multi-modal learning architecture in the presence of imperfect modalities.
• We introduce deep fusion architecture for object detection which processes each modality using the separate convolutional neural network (CNN) and constructs the joint feature map by combining the intermediate features from the CNNs.