2024

Debiased Multimodal Understanding for Human Language Sequences

Xu, Zhi, Yang, Dingkang, Li, Mingcheng et al.

Understand

Human multimodal language understanding (MLU) is an indispensable component of expression analysis (e.g., sentiment or humor) from heterogeneous modalities, including visual postures, linguistic contents, and acoustic behaviours.

Existing works invariably focus on designing sophisticated structures or fusion strategies to achieve impressive improvements.
Unfortunately, they all suffer from the subject variation problem due to data distribution discrepancies among subjects.
Concretely, MLU models are easily misled by distinct subjects with different expression customs and characteristics in the training data to learn subject-specific spurious correlations, limiting performance and generalizability across new subjects.

Built on

Causal reasoning from meta-reinforcement learning
Original
Dasgupta, I.; Wang, J.; Chiappa, S.; Mitrovic, J.; Ortega, P.; Raposo, D.; Hughes, E.; Battaglia, P.; Botvinick, M.; and Kurth-Nelson, Z. 2019 · 1901
Earlier work this paper cites.
Ur-funny: A multimodal language dataset for understanding humor
Original
Hasan, M. K.; Rahman, W.; Zadeh, A.; Zhong, J.; Tanveer, M. I.; Morency, L.-P.; et al. 2019 · 1904
Earlier work this paper cites.
Bounded rationality
Jones, B. D. 1999 · 1999
Earlier work this paper cites.
Models, reasoning and inference
Pearl, J.; et al. 2000 · 2000
Earlier work this paper cites.
Counterfactually-augmented SNLI training data does not yield better generalization than unaugmented data
Original
Huang, W.; Liu, H.; and Bowman, S. R. 2020 · 2010
Earlier work this paper cites.
Scalable k-means++
Original
Bahmani, B.; Moseley, B.; Vattani, A.; Kumar, R.; and Vassilvitskii, S. 2012 · 2012
Earlier work this paper cites.
COVAREP—A collaborative voice analysis repository for speech technologies
Degottex, G.; Kane, J.; Drugman, T.; Raitio, T.; and Scherer, S. 2014 · 2014
Earlier work this paper cites.
Glove: Global vectors for word representation
Pennington, J.; Socher, R.; and Manning, C. D. 2014 · 2014
Earlier work this paper cites.
Openface: an open source facial behavior analysis toolkit
Baltrušaitis, T.; Robinson, P.; and Morency, L.-P. 2016 · 2016
Earlier work this paper cites.
Causal inference in statistics: A primer
Glymour, M.; Pearl, J.; and Jewell, N. P. 2016 · 2016
Earlier work this paper cites.
Gaussian error linear units (gelus)
Original
Hendrycks, D.; and Gimpel, K. 2016 · 2016
Earlier work this paper cites.
Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos
Original
Zadeh, A.; Zellers, R.; Pincus, E.; and Morency, L.-P. 2016 · 2016
Earlier work this paper cites.
Facial expression analysis
iMotions. 2017 · 2017
Earlier work this paper cites.
Tensor fusion network for multimodal sentiment analysis
Original
Zadeh, A.; Chen, M.; Poria, S.; Cambria, E.; and Morency, L.-P. 2017 · 2017
Earlier work this paper cites.

Similar

Efficient low-rank multimodal fusion with modality-specific factors
Original
Liu, Z.; Shen, Y.; Lakshminarasimhan, V. B.; Liang, P. P.; Zadeh, A.; and Morency, L.-P. 2018 · 2018
Cited alongside, same era.
Learning factorized multimodal representations
Original
Tsai, Y.-H. H.; Liang, P. P.; Zadeh, A.; Morency, L.-P.; and Salakhutdinov, R. 2018 · 2018
Cited alongside, same era.
Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph
Zadeh, A.; and Pu, P. 2018 · 2018
Cited alongside, same era.
Found in translation: Learning robust joint representations by cyclic translations between modalities
Pham, H.; Liang, P. P.; Manzini, T.; Morency, L.-P.; and Póczos, B. 2019 · 2019
Cited alongside, same era.
Multimodal transformer for unaligned multimodal language sequences
Tsai, Y.-H. H.; Bai, S.; Liang, P. P.; Kolter, J. Z.; Morency, L.-P.; and Salakhutdinov, R. 2019 · 2019
Cited alongside, same era.
Words can shift: Dynamically adjusting word representations using nonverbal behaviors
Wang, Y.; Shen, Y.; Liu, Z.; Liang, P. P.; Zadeh, A.; and Morency, L.-P. 2019 · 2019
Cited alongside, same era.
MISA: Modality-invariant and-specific representations for multimodal sentiment analysis
Devamanyu, H.; Roger, Z.; and Soujanya, P. 2020 · 2020
Cited alongside, same era.
Deep facial expression recognition: A survey
Li, S.; and Deng, W. 2020 · 2020
Cited alongside, same era.
Integrating multimodal information in large pretrained transformers
Rahman, W.; Hasan, M. K.; Lee, S.; Zadeh, A.; Mao, C.; Morency, L.-P.; and Hoque, E. 2020 · 2020
Cited alongside, same era.
Unbiased scene graph generation from biased training
Tang, K.; Niu, Y.; Huang, J.; Shi, J.; and Zhang, H. 2020 · 2020
Cited alongside, same era.
Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis
Original
Han, W.; Chen, H.; and Poria, S. 2021 · 2021
Cited alongside, same era.
Progressive Modality Reinforcement for Human Multimodal Emotion Recognition From Unaligned Multimodal Sequences
Lv, F.; Chen, X.; Huang, Y.; Duan, L.; and Lin, G. 2021 · 2021
Cited alongside, same era.

Then

A Text-Centered Shared-Private Framework via Cross-Modal Prediction for Multimodal Sentiment Analysis
Wu, Y.; Lin, Z.; Zhao, Y.; Qin, B.; and Zhu, L.-N. 2021 · 2021
Later among the works it cites.
Deconfounded image captioning: A causal retrospect
Yang, X.; Zhang, H.; and Cai, J. 2021 · 2021
Later among the works it cites.
Causal attention for vision-language tasks
Yang, X.; Zhang, H.; Qi, G.; and Cai, J. 2021 · 2021
Later among the works it cites.
Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis
Yu, W.; Xu, H.; Yuan, Z.; and Wu, J. 2021 · 2021
Later among the works it cites.
De-biasing distantly supervised named entity recognition via causal intervention
Original
Zhang, W.; Lin, H.; Han, X.; and Sun, L. 2021 · 2021
Later among the works it cites.
Causal intervention for subject-deconfounded facial action unit recognition
Chen, Y.; Chen, D.; Wang, T.; Wang, Y.; and Liang, Y. 2022 · 2022
Later among the works it cites.
A causal debiasing framework for unsupervised salient object detection
Lin, X.; Wu, Z.; Chen, G.; Li, G.; and Yu, Y. 2022 · 2022
Later among the works it cites.
CubeMLP: An MLP-based model for multimodal sentiment analysis and depression estimation
Sun, H.; Wang, H.; Liu, J.; Chen, Y.-W.; and Lin, L. 2022 · 2022
Later among the works it cites.
Debiasing NLU models via causal intervention and counterfactual reasoning
Tian, B.; Cao, Y.; Zhang, Y.; and Xing, C. 2022 · 2022
Later among the works it cites.
Text-oriented Modality Reinforcement Network for Multimodal Sentiment Analysis from Unaligned Multimodal Sequences
Original
Lei, Y.; Yang, D.; Li, M.; Wang, S.; Chen, J.; and Zhang, L. 2023 · 2023
Later among the works it cites.
Towards Robust Multimodal Sentiment Analysis under Uncertain Signal Missing
Li, M.; Yang, D.; and Zhang, L. 2023 · 2023
Later among the works it cites.
Decoupled Multimodal Distilling for Emotion Recognition
Li, Y.; Wang, Y.; and Cui, Z. 2023 · 2023
Later among the works it cites.
Show, attend and tell: Neural image caption generation with visual attention
Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhudinov, R.; Zemel, R.; and Bengio, Y. 2015 · 2057
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…