2023

FER-former: Multi-modal Transformer for Facial Expression Recognition

Li, Yande, Wang, Mingjie, Gong, Minglun et al.

Understand

The ever-increasing demands for intuitive interactions in Virtual Reality has triggered a boom in the realm of Facial Expression Recognition (FER).

To address the limitations in existing approaches (e.g., narrow receptive fields and homogenous supervisory signals) and further cement the capacity of FER tools, a novel multifarious supervision-steering Transformer for FER in the wild is proposed in this paper.
Referred as FER-former, our approach features multi-granularity embedding integration, hybrid self-attention scheme, and heterogeneous domain-steering supervision.
In specific, to dig deep into the merits of the combination of features provided by prevailing CNNs and Transformers, a hybrid stem is designed to cascade two types of learning paradigms simultaneously.

Built on

L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.”
2008
Earlier work this paper cites.
M. Valstar, M. Pantic
2010
Earlier work this paper cites.
P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,” in
2010
Earlier work this paper cites.
G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. PietikäInen, “Facial expression recognition from near-infrared videos,”
2011
Earlier work this paper cites.
A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark,” in
2011
Earlier work this paper cites.
S. E. Kahou, C. Pal, X. Bouthillier, P. Froumenty, Ç. Gülçehre, R. Memisevic, P. Vincent, A. Courville, Y. Bengio, R. C. Ferrari
2013
Earlier work this paper cites.
Y. Tang, “Deep learning using linear support vector machines,”
Original
2013
Earlier work this paper cites.
I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee
2013
Earlier work this paper cites.
Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, “Ms-celeb-1m: A dataset and benchmark for large-scale face recognition,” in
2016
Earlier work this paper cites.
E. Barsoum, C. Zhang, C. Canton Ferrer, and Z. Zhang, “Training deep networks for facial expression recognition with crowd-sourced label distribution,” in
2016
Earlier work this paper cites.
K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,”
2016
Earlier work this paper cites.
Y. Gu, Z. Zhong, S. Wu, and Y. Xu, “Enlarging effective receptive field of convolutional neural networks for better semantic segmentation,” in
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”
2017
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”
Original
2018
Earlier work this paper cites.
D. Acharya, Z. Huang, D. Pani Paudel, and L. Van Gool, “Covariance pooling for facial expression recognition,” in
2018
Earlier work this paper cites.
J. Cai, Z. Meng, A. S. Khan, Z. Li, J. O’Reilly, and Y. Tong, “Island loss for learning discriminative features in facial expression recognition,” in
2018
Earlier work this paper cites.
J. Zeng, S. Shan, and X. Chen, “Facial expression recognition with inconsistently annotated datasets,” in
2018
Earlier work this paper cites.
Y. Li, J. Zeng, S. Shan, and X. Chen, “Occlusion aware facial expression recognition using CNN with attention mechanism,”
2019
Earlier work this paper cites.
Y. Han, B. Tang, and L. Deng, “An enhanced convolutional neural network with enlarged receptive fields for fault diagnosis of planetary gearboxes,”
2019
Earlier work this paper cites.
B. Pan, S. Wang, and B. Xia, “Occluded facial expression recognition enhanced through privileged information,” in
2019
Earlier work this paper cites.
J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in
2019
Earlier work this paper cites.
S. Li and W. Deng, “Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition,”
2019
Earlier work this paper cites.

Similar

L. Yang, Y. Tian, Y. Song, N. Yang, K. Ma, and L. Xie, “A novel feature separation model exchange-gan for facial expression recognition,”
2020
Cited alongside, same era.
K. Wang, X. Peng, J. Yang, D. Meng, and Y. Qiao, “Region attention networks for pose and occlusion robust facial expression recognition,”
2020
Cited alongside, same era.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly
Original
2020
Cited alongside, same era.
S. Chen, J. Wang, Y. Chen, Z. Shi, X. Geng, and Y. Rui, “Label distribution learning on auxiliary label space graphs for facial expression recognition,” in
2020
Cited alongside, same era.
K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” in
2020
Cited alongside, same era.
B. Xia and S. Wang, “Occluded facial expression recognition with step-wise assistance from unpaired non-occluded images,” in
2020
Cited alongside, same era.
F. Zhang, T. Zhang, Q. Mao, and C. Xu, “Geometry guided pose-invariant facial expression recognition,”
2020
Cited alongside, same era.
J. Wang, L. Song, Z. Li, H. Sun, J. Sun, and N. Zheng, “End-to-end object detection with fully convolutional network,” in
2021
Cited alongside, same era.
H. Sun, X. Zheng, and X. Lu, “A supervised segmentation network for hyperspectral image classification,”
2021
Cited alongside, same era.
Z. Zhao, Q. Liu, and S. Wang, “Learning deep global multi-scale and local attention features for facial expression recognition in the wild,”
2021
Cited alongside, same era.
F. Xue, Q. Wang, and G. Guo, “Transfer: Learning relation-aware facial expression representations with transformers,” in
2021
Cited alongside, same era.
Y. Chen and J. Joo, “Understanding and mitigating annotation bias in facial expression recognition,” in
2021
Cited alongside, same era.

Then

Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao, “Simvlm: Simple visual language model pretraining with weak supervision,”
Original
2021
Later among the works it cites.
R. Hu and A. Singh, “Unit: Multimodal multitask learning with a unified transformer,” in
2021
Later among the works it cites.
W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in
2021
Later among the works it cites.
P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao, “Vinvl: Revisiting visual representations in vision-language models,” in
2021
Later among the works it cites.
C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in
2021
Later among the works it cites.
H. Li, N. Wang, X. Ding, X. Yang, and X. Gao, “Adaptively learning facial expression representation via cf labels and distillation,”
2021
Later among the works it cites.
W. Zhang, X. Ji, K. Chen, Y. Ding, and C. Fan, “Learning a facial expression embedding disentangled from identity,” in
2021
Later among the works it cites.
Y. Li, Y. Lu, M. Gong, L. Liu, and L. Zhao, “Dual-channel feature disentanglement for identity-invariant facial expression recognition,”
2022
Later among the works it cites.
Q. Zhou, X. Wu, S. Zhang, B. Kang, Z. Ge, and L. J. Latecki, “Contextual ensemble network for semantic segmentation,”
2022
Later among the works it cites.
J. Gu, H. Kwon, D. Wang, W. Ye, M. Li, Y.-H. Chen, L. Lai, V. Chandra, and D. Z. Pan, “Multi-scale high-resolution vision transformer for semantic segmentation,” in
2022
Later among the works it cites.
L. Sun, G. Zhao, Y. Zheng, and Z. Wu, “Spectral–spatial feature tokenization transformer for hyperspectral image classification,”
2022
Later among the works it cites.
G. Han, J. Ma, S. Huang, L. Chen, and S.-F. Chang, “Few-shot object detection with fully cross-transformer,” in
2022
Later among the works it cites.
Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li, “Maxim: Multi-axis mlp for image processing,” in
2022
Later among the works it cites.
J. Shao, Z. Wu, Y. Luo, S. Huang, X. Pu, and Y. Ren, “Self-paced label distribution learning for in-the-wild facial expression recognition,” in
2022
Later among the works it cites.
H. Li, N. Wang, X. Yang, X. Wang, and X. Gao, “Towards semi-supervised deep facial expression recognition with an adaptive confidence margin,” in
2022
Later among the works it cites.
L. Wang, G. Jia, N. Jiang, H. Wu, and J. Yang, “Ease: Robust facial expression recognition via emotion ambiguity-sensitive cooperative networks,” in
2022
Later among the works it cites.
C. Wang, M. Chai, M. He, D. Chen, and J. Liao, “Clip-nerf: Text-and-image driven manipulation of neural radiance fields,” in
2022
Later among the works it cites.
J. Guo, K. Han, H. Wu, Y. Tang, X. Chen, Y. Wang, and C. Xu, “Cmt: Convolutional neural networks meet vision transformers,” in
2022
Later among the works it cites.
A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela, “Flava: A foundational language and vision alignment model,” in
2022
Later among the works it cites.
J. Jiang and W. Deng, “Disentangling identity and pose for facial expression recognition,”
2022
Later among the works it cites.
H. Li, N. Wang, X. Yang, and X. Gao, “Crs-cont: A well-trained general encoder for facial expression analysis,”
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…