Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion

Understand

We propose an audio-visual spatial-temporal deep neural network with: (1) a visual block containing a pretrained 2D-CNN followed by a temporal convolutional network (TCN); (2) an aural block containing several parallel TCNs; and (3) a leader-follower attentive fusion block combining the audio-visual information.

• The TCN with large history coverage enables our model to exploit spatial-temporal information within a much larger window length (i.e., 300) than that from the baseline and state-of-the-art methods (i.e., 36 or 48).
• The fusion block emphasizes the visual modality while exploits the noisy aural modality using the inter-modality attention mechanism.
• To make full use of the data and alleviate over-fitting, cross-validation is carried out on the training and validation set.